Patent Publication Number: US-2023147412-A1

Title: Systems and methods for authoring immersive haptic experience using spectral centroid

Description:
TECHNICAL FIELD 
     This application is a continuation of PCT application PCT/EP2021/069371, which was filed on Jul. 12, 2021, and that PCT application claims the benefit of the filing date of U.S. provisional application no. 63/050,834, filed on 12 Jul. 2020, and the disclosures of each of these filings are incorporated herein by reference in their respective entireties. 
     This disclosure relates to a haptic processing system for generation of haptic data using audio signal analysis. More specifically, the descriptions provided herein relate to analyzing audio signals using the spectral centroid and authoring the audio signal to produce haptic data. 
    
    
     BACKGROUND 
     Haptic usually refers to a sense of touch or perception provided to a user as a feedback force or vibration. An electronic computing device with haptic feedback can substantially improve human computer interface. The feedback force provides a sense of perception of touch and feel, which can enhance the user experience. With technological advancement, user interfaces integrated with haptics. The haptic feedback provided by the different types of devices are distinguishable, providing a sense of different feel and touch. 
     A complex process of filtering, transformation and editing is required to efficiently convert audio signal into haptic data to provide a fulfilling user experience. To provide a fulfilling user experience, the audio signal is converted into haptic data which then can be authored and enhanced. The haptic experience is delivered using haptic actuators such as Linear Resonant Actuators (LRA), Wide Band or High Definition actuators, piezo-electric actuators etc. The delivery of the haptic experience is dependent on the audio to haptic conversion of the signal, the response characteristics of the haptic actuator, device specific data, among other factors. Therefore, a proper matching of actuator type and its response characteristics is required to augment user experience. 
     US application 20200379570 provides a haptic system and method for defining haptic patterns that include both haptic events and audio events. The haptic pattern can be called by an application programming interface, which has haptic mapping functionality that has a haptic experience mapping functionality that generates a same, or similar, haptic experience on different manufacturers or models of electronic devices having different haptic hardware. The prior art provides a method of mapping haptic functionality on different devices for similar experience. However, the disclosed descriptions do not provide authoring the audio content, instead, certain embodiments provide a haptic pattern that has embedded haptic events within the audio events. 
     US application 20210181851 relates to an adaptive haptic signal generating device. The adaptive signal generating device includes a frequency analysis unit for converting and analyzing a received audio signal in a frequency domain. A frequency equalizer unit allows the adaptive haptic signal generating device to suppress or amplify a specific frequency/frequencies in the frequency domain. A haptic event extraction unit extracts a haptic event based on the suppressed or amplified frequencies. The device then generates a haptic signal corresponding to the haptic event signal. A control unit counts the occurrence of the extracted haptic event signal for each frequency. It then increases the frequency gain of a frequency that has been generated more than a specific predefined number of times. This provides a method of generating haptic events in the frequency domain but does not provide analysis and authoring of haptic output based at least on device characteristics. 
     US application 20210110681 provides a method of authoring the audio signal into a haptic signal using a filter bank and harmonic percussive source separation but uses a different approach of time amplitude frequency analysis. In contrast, the current application uses a novel approach of calculating the spectral centroid to author audio signals into haptic output. 
     U.S. Pat. No. 10,467,870 provides a haptic conversion system that analyzes an audio signal, generates haptic signals based on the analysis of the audio signal. The haptic conversion system then plays the generated haptic signals through one or more actuators to produce haptic effects. The haptic conversion system maps the generated haptic signals to the different actuators based on one or more audio characteristics of the audio signal. This discloses a conversion of an audio signal into a haptic signal but fails to disclose authoring the audio signal based at least on the device parameters. 
     U.S. Pat. No. 9,448,626 discloses a haptic conversion system that intercepts frames of audio data to convert the frames into a haptic signal and plays the created haptic signal through an actuator to produce haptic effects. The haptic signal is based on a maximum value of each audio data frame. The maximum value defines the magnitude of the haptic signal. The haptic signal is applied to the actuator to generate the one or more haptic effects. Embodiments described herein provide a haptic conversion system but do not disclose authoring of the audio content for an immersive haptic experience. 
     U.S. Pat. No. 9,092,059 discloses a haptic conversion system that receives multiple audio streams of an audio signal. The haptic conversion system then evaluates each stream to determine if at least one parameter of the one or more parameters indicates that stream is to be converted into a haptic effect. The haptic conversion system then identifies whether one or more streams that include at least one parameter are to be converted into the haptic effect. The haptic conversion system further generates, for the identified streams, a haptic signal based on each corresponding stream and sends, for the identified streams, each haptic signal to an actuator to generate a haptic effect. Certain embodiments described herein do not disclose authoring of the converted haptic data from the audio signal. 
     All cited prior art fails to disclose the novel and unique features of authoring the haptic output. The method and system disclosed provide for: analysis of the audio signal, conversion of audio signal into haptic output, authoring of the converted audio signal into haptic signal based on device characteristics and the embedded actuator characteristics and transforming the frequency bands into the haptic perceptual bandwidth for an immersive haptic experience. 
     SUMMARY 
     Provided herein is a computer implemented method of editing and transforming an audio signal into a haptic data to provide an immersive haptic experience. The computer implemented method receives the audio signal at a preprocessor module to determine a peak to peak amplitude of the audio signal for an audio frame having a fixed number of sampled audio data. In embodiments, the audio frame may include one or more audio packets. Alternatively, the audio signal may be processed based on a fixed or variable window size comprising audio packets or audio sampled data. The computer implemented method performs a fast Fourier transform to derive the frequency distribution of the preprocessed audio signal. The fast Fourier transform comprises an array of time amplitude values, an array of time frequency values, and an array of time amplitude frequency values. Subsequently, the computer implemented method calculates the weighted average mean of the derived frequency distribution to calculate or derive a spectral centroid of the sampled audio data for a fixed or variable size window. The sampled audio data is then replaced with a new sampled audio data and the spectral centroid is calculated. Thereafter, the computer implemented method calculates a new spectral centroid of the audio signal to produce an array of the spectral centroid. The array of the spectral centroid is provided to a user interface to modify the array of spectral centroid for the immersive haptic experience. Finally, the computer implemented method produces a computer readable file that can be parsed by the resynthesis module on an electronic computing device to provide an immersive haptic experience. 
     In a variation of this implementation, the array of the spectral centroid values may include an array of time amplitude values and an array of time frequency values, which can be modified using the user interface. The modification of the array of spectral centroid values is based at least on a device specific information and an actuator specific information. In embodiments, the device specific information includes device mass, device type, device operating characteristics and actuator specific characteristics embedded in the device. 
     The spectral centroid of the preprocessed audio signal is calculated by adding the sum of the spectral energy of each frequency multiplied by the average value of each frequency band over the sum of spectral energy for all the frequency bands. 
     In another implementation, a computer implemented method of authoring and transforming an audio signal into a haptic output to provide immersive haptic experience comprises the steps of: receiving the audio signal at a preprocessor for a fixed number of audio samples; applying harmonic percussive source separation to pre-processed audio samples, wherein the harmonic percussive source separation comprises a harmonic spectrogram and a percussive spectrogram; calculating an array of time amplitude values and an array of time frequency values for the harmonic spectrogram and the percussive spectrogram; providing the array of the spectral centroid, the array of time amplitude values and the array of time frequency values to a user interface to modify at least one of the array of values of the harmonic spectrogram and/or the percussive spectrogram; transforming authored array of spectral centroid, the array of time amplitude values and the array of time frequency values to fit into a haptic perceptual bandwidth, and creating a computer readable file. 
     In one variation of this implementation, the harmonic percussive source separation further includes a residual spectrogram, which is obtained by subtracting the harmonic spectrogram and the percussive spectrogram from the transformed harmonic percussive source separation spectrogram. 
     In one variation of this implementation, the transformation of the authored array of spectral centroid, the array of time amplitude values and the array of time frequency values is based at least on a device specific information and an actuator specific information. In embodiments, the spectral centroid for the harmonic component and the percussive component is calculated by adding the sum of the spectral energy multiplied by the average value of each frequency band over the sum of spectral energy of all the frequency bands. 
     A haptic authoring system for converting an audio signal into a computer readable haptic file, the computer readable file when executed by a processor causes the haptic processing system to produce an immersive haptic experience on the associated electronic computing device, the haptic authoring system comprising: a preprocessor module configured to an audio analysis module, wherein the audio analysis module receives a preprocessed audio signal and converts the preprocessed audio signal into an array of time amplitude values and an array of time frequency values, and wherein the audio analysis module calculates the spectral centroid of the array of time frequency values; an user interface for modifying the array of time amplitude values, the array of time frequency values and the array of time frequency values; a transformation module for transforming an authored array of time amplitude values, an authored array of time frequency values and an authored array of time frequency values to fit into a haptic perceptual bandwidth, and an aggregation and file management module for converting an array of transformed time amplitude values, an array of transformed time frequency values and an array of transformed time frequency values into the computer readable haptic file. 
     The audio analysis module calculates the spectral centroid by adding the sum of the spectral energy of each frequency multiplied by the average value of each frequency band over the sum of spectral energy for all the frequency bands. 
     The transformation module may transform the harmonic spectrogram and the percussive spectrogram based at least on a device specific information, including device mass, device type, device operating characteristics and an actuator specific characteristics embedded in the device. 
     A haptic authoring system for converting an audio signal into a computer readable haptic file, the computer readable file when executed by a processor causes the haptic authoring system to produce an immersive haptic experience on the associated electronic computing device, the haptic authoring system comprising: a preprocessor module configured to an audio analysis module, wherein the audio analysis module receives a preprocessed audio signal and applies a harmonic percussive source separation to the preprocessed audio signal to determine a harmonic spectrogram and a percussive spectrogram, and wherein the audio analysis module calculates the spectral centroid of the harmonic spectrogram and the percussive spectrogram; an user interface for modifying the harmonic spectrogram and the percussive spectrogram, wherein the harmonic spectrogram comprises an array of time amplitude values and an array of time frequency values and the percussive spectrogram comprises an array of time frequency values and an array of impulse sequence; a transformation module for transforming an authored harmonic spectrogram and an authored percussive spectrogram to fit into a haptic perceptual bandwidth, and an aggregation and file management module for converting a transformed harmonic spectrogram and a transformed percussive spectrogram into the computer readable haptic file. 
     The haptic authoring system includes the audio analysis module, which calculates the spectral centroid by adding the sum of the spectral energy of each frequency multiplied by the average value of each frequency band over the sum of spectral energy for all the frequency bands. 
     The transformation module of the haptic authoring system transforms the harmonic spectrogram and the percussive spectrogram based at least on device specific information, including device mass, device type, device operating characteristics and actuator specific characteristics embedded in the device. 
     Disclosed is a method and system of authoring an audio signal to produce an immersive haptic experience. The method preprocesses the audio signal in a preprocessor module. The preprocessed signal is passed to an audio analysis module. The audio analysis module processes the preprocessed audio signal for a fixed time to produce (a) an array of time amplitude values, and (b) an array of spectral centroid (i.e. frequency) values. To produce the array of time amplitude values, the preprocessed audio signal is also passed to an envelope approximation and smoothing module to (a) approximate the preprocessed audio signal into time amplitude values, and (b) smoothen the approximated time amplitude values. The array of time amplitude values is passed to a breakpoint reduction module. The breakpoint reduction module reduces the array of time amplitude values based on a linear approximation of a time sequence. The preprocessed audio signal is also passed to a DC Offset module. The purpose of the DC offset module is to add a small value offset to avoid spectral centroid from rising spuriously during segments, where the audio signal is silent. During absolute or near silence, the audio signal has small amplitude values, which allows high frequencies to dominate during silence. The presence of high frequencies is nor desirable in audio segments of silence. The output of the DC offset is provided to the central tracker module, which calculates the spectral centroid of the audio signal or segments of the audio signal processed in a specific window or a block of audio packets. The spectral centroid is the centre of mass of the block-wise Fourier transformed power spectrum of the audio signal, which provides the dominant frequency of the audio signal for each windowed signal block processed this way. By identifying the dominant frequency for each point in time (i.e. each frame comprising audio data blocks or each window of audio data), the oscillator driving a haptic actuator can be tuned to the spectral centroid value at each point in time, providing a good haptic experience. The novel and unique technique of the spectral centroid tracking is synchronized with the amplitude tracking to provide a better haptic experience. 
     In some embodiments, other statistics may be utilized such as a spectral bandwidth, a spectral skewness, a spectral flatness, spectral kurtosis, a spectral contrast and a spectral roll-off. 
     In some embodiments, the time amplitude values are arrays of time amplitude values for different frequency bands. 
     In parallel, the spectral centroid is calculated from the same preprocessed audio signal for which the time-amplitude values were derived. The preprocessed audio signal is passed into a Fast Fourier Transform to convert the time-domain signal into a frequency domain signal. In some embodiments, the Short Time Fourier Transform may be performed to convert the time domain audio signal into a frequency domain signal. 
     For a fixed time, a particular number of samples are calculated. The spectral centroid of this particular number of samples is calculated by using a weighted average method. The average frequency of each frequency interval is calculated; the average frequency is multiplied by the spectral energy of that frequency interval to calculate the spectral energy distribution for that frequency band. Similarly, the spectral energy distribution for all frequency bands is calculated. The energy frequency distribution of all the frequency bands is divided by the sum of the average frequency of all the frequency bands to arrive at the spectral centroid. To calculate the array of spectral centroid values, a fixed number of samples are removed and an equal number of samples are added. For example if the buffer size of the frame of the preprocessed audio signal is  1024  samples, each iteration may replace  128  samples and introduce  128  new samples. In this example, the array of spectral centroid values may be obtained. 
     In some embodiments, the iteration is performed for the same and fixed number of samples for calculating the time amplitude values and spectral centroid values. 
     In some embodiments, the iteration is performed for the same and fixed number of audio frames for calculating the time amplitude values and the spectral centroid values. 
     In some embodiments, the iteration is performed for the same and fixed window size for calculating the time amplitude values and the spectral centroid values. 
     The array of time amplitude values and the array of spectral centroid values are passed to a user interface of the authoring tool for editing and/or modifying and/or appending the array of time amplitude values and the array of spectral centroid values. The transformation module receives the authored array of time amplitude values, the authored array of time amplitude values and the authored array of the spectral centroid values to determine if the authored array of time amplitude values, the authored array of time amplitude values and the authored array of the spectral centroid values can fit within a haptic perceptual bandwidth for each frequency bands. The haptic perceptual bandwidth is the range of frequencies that are above which an actuator embedded in an electronic computing device can reproduce the haptic experience that can be experienced by humans. If the authored array of time amplitude values, the authored array of time amplitude values and the authored array of the spectral centroid values can fit within a haptic perceptual bandwidth for each of the frequency bands, then the authored array of time amplitude values, the authored array of time amplitude values and the authored array of the spectral centroid values is passed to an aggregation and file management module. Otherwise, the authored array of time amplitude values, the authored array of time amplitude values and the authored array of the spectral centroid values are transformed into a transformed array of time amplitude values, a transformed array of time frequency values and a transformed array of the spectral centroid values by implementing a computer implemented algorithm. The computer implemented algorithm performs steps: determining the rank of each frequency band as provided in the user interface. Alternatively, the computer implemented algorithm may calculate the rank of each frequency band based on the spectral energy of each frequency band and its distance from the combined resonant frequency of the electronic computing device having an embedded actuator. Subsequently, the computer implemented algorithm may shift each frequency band and eliminate some of the frequency bands that cannot fit as defined in the algorithm. The transformed array of time amplitude values, the transformed array of time frequency values, and the transformed array of the spectral centroid values are passed to an aggregation and file management module, which converts the transformed array of time amplitude values, the transformed array of time frequency values, and the transformed array of spectral centroid values into a computer readable file format. 
     Subsequently, the transformed array of time amplitude values, the transformed array of time frequency values, and the transformed array of spectral centroid values are saved in a computer readable haptic file that can be parsed and processed by the resynthesis module. 
     In some embodiments, the transformed array of time amplitude values, the transformed array of time frequency values, and the transformed array of spectral centroid values are directly passed to the resynthesis module for generating haptic output. 
     In some embodiments, the processing of authoring the array of time amplitude values, the array of time frequency values, and the array of spectral centroid values happen in real time by applying the deep learning algorithm. 
     The resynthesis module uses the transformed array of time amplitude values, the transformed array of time frequency values, and the transformed array of spectral centroid values to generate haptic output in one or more actuators. In one variation of this implementation, the transformed array of time amplitude values are used to set the amplitude of one or more actuators and the transformed array of spectral centroid values are used to set the center frequency of the one or more actuators for providing an immersive haptic experience. 
     In one embodiment, the preprocessed audio signal may be passed to a filter bank. The filter bank may separate the audio signal into different frequency bands and process each frequency band to produce an array of time amplitude values, an array of time frequency values, and an array of spectral centroid values. 
     In another embodiment, the preprocessed audio signal may be processed separately using a Harmonic Percussive Source Separation (HPSS) module to produce a harmonic spectrogram, a percussive spectrogram, and a residual spectrogram. The harmonic module produces a harmonic spectrogram, Likewise, the percussive module produces the percussive spectrogram and the residual module produces a residual spectrogram. The HPSS module may perform the Fast Fourier Transform (FFT) or the Short Time Fast Transform (STFT) of the received preprocessed audio signal to convert the time domain signal to the frequency domain signal. The FFT or STFT produces the power spectrogram ,which is utilized to produce the harmonic spectrogram and the percussive spectrogram. The harmonic module creates a harmonic spectrogram by median filtering. Likewise, the percussive spectrogram is obtained by filtering the power spectrogram. The audio signal received after filtering from the percussive module is passed to the centroid tracker to calculate the spectral centroid of the percussive spectrogram. The spectral centroid is passed to the envelope approximation and smoothing module. The spectral centroid provides the measure of the main or the dominant frequency of the audio signal. 
     In embodiments, the spectral centroid may be calculated for a recursively fixed number of audio samples resulting in the array of spectral centroid values for a fixed time. 
     To produce an array of time amplitude frequency values, the harmonic spectrogram is passed to an envelope approximation and smoothing module to (a) approximate the preprocessed audio signal into an time amplitude frequency values, and (b) smoothen the approximated the time amplitude frequency values. The array of time amplitude frequency values is then passed to a breakpoint reduction module. The breakpoint reduction module reduces the array of time amplitude frequency values based on a linear approximation of a time sequence. Finally, the array of time amplitude frequency values is passed to the amplitude envelope module for extracting an array of time amplitude values from the harmonic spectrogram. 
     The harmonic spectrogram is also passed to an centroid tracker for calculating the spectral centroid of the array of time amplitude frequency values, Subsequently, the spectral centroid values are passed to an envelope approximation and smoothing module to (a) approximate the preprocessed audio signal into an time amplitude frequency values, and (b) smoothen the time amplitude frequency values. The array of time amplitude frequency values is then passed to a breakpoint reduction module. The breakpoint reduction module reduces the array of time amplitude frequency values based on a linear approximation of a time sequence. Finally, the array of time amplitude frequency values is passed to the frequency envelope module for extracting an array of time frequency values, an array of time amplitude values, and an array of spectral centroid values for the harmonic spectrogram. 
     The power spectrogram is passed to the percussive module that extracts the percussive spectrogram by filtering. The percussive spectrogram comprises time amplitude frequency values. The percussive spectrogram is then passed to a centroid tracker, which calculates the array of spectral centroid of the audio signal. To produce an array of time amplitude values, the array of spectral centroid values are passed to an envelope approximation and smoothing module to (a) approximate the preprocessed audio signal into an time amplitude values, and (b) smoothen the approximated time amplitude values. The array of time amplitude values is then passed to a breakpoint reduction module. The breakpoint reduction module reduces the array of amplitude time values based on a linear approximation of a time sequence. Finally, the array of time amplitude time is passed to an amplitude envelope module for extracting a reduced array of time amplitude values. Simultaneously, the percussive module passes the percussive spectrogram to the transient module. The transient module detects the presence of transients in the percussive spectrogram and passes the transients to an impulse sequence module to create an array of impulse sequences. 
     In some embodiments, the transients detected may comprise one or more impulses. The one or more impulses may form the array of impulse sequences, which comprise an array of time amplitude values and/or an array of time frequency files. 
     The residual module processes the residual spectrogram. The residual spectrogram is passed to an envelope approximation and smoothing module to (a) approximate the preprocessed audio signal into an array of time amplitude frequency values, and (b) smoothen the approximated signal into an array of time amplitude frequency values. The array of time amplitude frequency values is then passed to a breakpoint reduction module. The breakpoint reduction module reduces the array of time amplitude frequency values based on a linear approximation of a time sequence. Finally, the array of time amplitude frequency values is passed to an amplitude envelope module for extracting an array of amplitude time values from the residual spectrogram. 
     In some embodiments, the spectral centroid of the received audio signal may be calculated for a predefined number of samples over a fixed time or for a fixed number of samples. 
     The time domain signal is converted into the frequency domain signal by implementing a Fast Fourier Transform (FFT) or a Short Time Fourier Transform (STFT) before performing the spectral centroid calculations. To calculate the spectral centroid of the preprocessed audio signal, the frequency spectrum is analyzed, for each frequency band, the average frequency is calculated, the average frequency of each band is multiplied by the spectral energy of that frequency band. The sum of product of average frequency and spectral energy of all frequency bands is calculated, which is divided by the sum of spectral energy of all frequency bands. After calculating the spectral centroid, the audio samples are left shifted, that is, removing the last predefined number of samples and replacing them with the same number of new audio samples. In this way an array of spectral centroid values is produced, which are provided to the authoring tool for editing and/or modifying the array of spectral centroid values. 
     The array of time amplitude values and the array of impulse sequences obtained by analysis of transients from the percussive spectrogram is provided to an impulse processing module. In addition, the impulse processing module also receives the array of time amplitude values and the array of time frequency values from the harmonic spectrogram. Additionally, the array of time amplitude frequency values from the residual module is also passed to an authoring tool for modifying/editing/appending the analyzed audio signal. 
     In some embodiments, the residual module is optional and only the harmonic module and the percussive module are used for analysis. The array of time amplitude values and the array of impulse sequences received from the percussive module; the array of time amplitude values and the array of time frequency values received from the harmonic module may be provided to an impulse processing module. 
     In some embodiments, the time amplitude frequency values from the residual module may be passed to the impulse processing module along with the audio signal from the harmonic module and the percussive module. 
     In some embodiments, the time amplitude frequency values from the residual module may be passed directly to the authoring tool. 
     The authoring tool includes a Graphical User Interface (GUI) to edit and/or modify and/or append the array of time amplitude values and the array of impulse sequences from the impulse sequence of the percussive module. In addition, the GUI also receives the array of time amplitude values and the array of time frequency values from the harmonic module. Further, the array of time amplitude frequency values from the residual module are also provided to the GUI editor. 
     In some embodiments, no modification is performed by the user through GUI to the arrays of time amplitude values, the array of impulse sequence from the percussive module and the array of time amplitude values and the array of time frequency values of the harmonic module. In this embodiment, no authoring of the analyzed audio signal received from the audio analysis module is required. In an alternate embodiment, the authoring of the analyzed audio signal received from the audio analysis module is performed automatically using deep learning algorithms. The trained deep learning algorithm continuously learns from ongoing data analysis. 
     In some embodiments, the authoring tool may be bypassed and no authoring of the analyzed audio signal may be performed. In some embodiments, the residual module may be ab sent. 
     The transformation module receives an authored array of time frequency values and an authored array of time amplitude values from the continuous stream; an authored array of time amplitude values and an authored array of impulse sequence from the impulse stream to determine if the authored array of time amplitude; the authored array of impulse sequence; the authored array of time frequency values; the authored array of time amplitude frequency values received from the continuous stream, the impulse stream, and the residual module (optional) can fit within a haptic perceptual bandwidth for each of the frequency bands. If the authored array of time amplitude; the authored array of impulse sequence; the authored array of frequency time values; the authored array of time amplitude frequency values from the continuous stream and the impulse stream can fit within a haptic perceptual bandwidth for each of the frequency bands, then the these authored array of values are passed to an aggregation and file management module. Otherwise, the authored array of time amplitude; the authored array of impulse sequence; the authored array of frequency time values; the authored array of amplitude frequency values is transformed into a transformed continuous stream and a transformed impulse stream by implementing an algorithm. The algorithm performs steps determining the rank of each of frequency bands or calculating the rank of each of frequency bands, shifting of frequency bands, and eliminating some of the frequency bands as per the implemented algorithm. The transformed array of time amplitude frequency values is passed to an aggregation and file management module, which converts the transformed array of values into a computer readable file format. Finally, the computer readable file format can be parsed by a resynthesis module having a synthesizer to generate a haptic output. 
     In some embodiments, the comparator uses statistical analysis and weighting to make informed decisions on the two streams of impulses, that is, the harmonic stream and the percussive stream to remove duplicates and/or overlaps. 
     In some embodiments, the comparator may use analytics and machine learning to predict and/or merge or remove duplicates during merging of the harmonic stream and the percussive stream. 
     In some embodiments, the slope of the impulse events generated from the harmonic spectrogram is utilized for generating and/or marking the impulse events. The slope calculated from the harmonic spectrogram is compared with a threshold value of the slope. If the gradient of the slope is greater than the threshold value then an impulse event is generated. 
     In some embodiments, the sharpness value is also recorded by measuring the slope steepness/gradient. The sharpness value is passed into the comparator. 
     In some embodiments, the impulse events generated from the percussive component are detected by the transient detector by comparing the values of a fast envelope follower and the slow envelope follower. 
     In some embodiments, the impulse events are generated from the percussive component using the transient detection algorithm. The impulse events are also generated in parallel by the harmonic component. In some embodiments, the comparator receives two sets of impulse signals (a) processed from the harmonic spectrogram and (b) processed from the percussive spectrogram are merged to form one impulse sequence. 
     In some embodiments, the output of the comparator with merged impulse events from the harmonic stream and the percussive stream are analyzed, the overlaps merged intelligently, and then ducking off the continuous envelope when an impulse event occurs. 
     In some embodiments, the merged impulse events may be directly provided to the transformation module and to the mixer for providing haptic experience. 
     In some embodiments, the merged impulse events may be provided to the authoring tool for editing and or modification before passing it to the transformation module and to the mixer for providing haptic experience. 
     In some embodiments, the user interface editor associated with the authoring tool  208  may receive an array of time-frequency values or the frequency envelope of the audio signal. 
     In one embodiment, the frequency envelope may be obtained by a spectral centroid. In another embodiment, the frequency envelope may be obtained from the percussive spectrogram and/or the harmonic spectrogram. 
     The frequency envelope generated by the spectral centroid is smoothed and breakpoint-reduced to reduce the frequency envelope. The frequency envelope is then passed out the user interface, for example, the user interface. The user interface displays the frequency envelope together with the amplitude envelope for the same set of audio samples or audio signals. In some embodiments, the user can view and adjust the frequency envelope values. In addition, the user can also edit the amplitude envelope values. The user interface associated with the authoring tool provides a novel method of dealing with audio silence. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG.  1    illustrates an overview of an operating environment of a haptic processing system in an embodiment; 
         FIG.  2    illustrates different modules of a haptic module in the embodiment; 
         FIG.  3    illustrates a haptic module operating in a distributed environment in another embodiment; 
         FIG.  4 A  illustrates the block diagram of an audio analysis module implementing a spectral centroid for converting an audio signal into a haptic output in an embodiment; 
         FIG.  4 B  illustrates the block diagram of an audio analysis module implementing a filter banks for converting an audio signal into a haptic output in another embodiment; 
         FIG.  5 A  illustrates the block diagram of an audio analysis module implementing a harmonic-percussive source separation for converting an audio signal into a haptic output in an embodiment; 
         FIG.  5 B  illustrates the block diagram of an audio analysis module implementing a harmonic-percussive source separation for converting an audio signal into a haptic output in another embodiment; 
         FIG.  5 C  illustrates the block diagram of the impulse processing module in an embodiment; 
         FIG.  5 D  illustrates the block diagram of an authoring tool for processing audio data in an embodiment; 
         FIG.  6    illustrates the block diagram of processing of haptic output for producing haptic experience in an embodiment; 
         FIG.  7    illustrates the block diagram of processing of the in another embodiment; 
         FIG.  8 A  illustrates the method of detecting an impulse in an audio stream using gradient approach in the embodiment; 
         FIG.  8 B  illustrates the method of merging the impulse in different audio signals using gradient approach in the embodiment; 
         FIG.  9    illustrates a graphical user interface of an authoring tool in the embodiment; 
         FIG.  10    illustrates a block diagram of a transformation module in the embodiment; 
         FIG.  11    illustrates an aggregation and file management module in the embodiment; 
         FIG.  12    illustrates a resynthesis module in the embodiment; 
         FIG.  13    illustrates the process of handling an audio signal having audio silence in an embodiment; 
         FIG.  14    illustrates a process of converting an audio signal into a computer readable haptic file in an embodiment; 
         FIG.  15    illustrates a process of implementing a filterbank analysis to an audio signal in an embodiment; and 
         FIG.  16    illustrates a process of implementing a harmonic percussive source separation analysis to an audio signal in an embodiment. 
     
    
    
     DETAILED DESCRIPTION 
     As used herein, the term “input audio signal”, “received signal”, “processed signal”, “audio signal” are intended to broadly encompass all types of audio signal including analog audio signal, digital audio signal, digital audio data, audio signal embedded in media programs including signal embedded in video or audio that can be rendered using a rendering device capable of reproducing any other type of audio or media program connected to a network or any electronic device operating independently. It also encompasses live media, linear media and interactive media programs such as music, games, online video games or any other type of streamed media program with embedded audio. Furthermore, these terms also include an array of time amplitude values, an array of time frequency values, an array of time amplitude frequency values, an array of impulse sequence values to substantiate the contextual meaning at different places. 
       FIG.  1    illustrates an overview of an operating environment of a haptic processing system in an embodiment. The operating environment of a haptic processing system  100 , an electronic computing device  102  connected to a cloud  140 , a server  160 , and a distributed system  150  via wired or wireless network . The operating environment  100  is exemplary and other variations may include different implementations with fewer or additional things. 
     The electronic computing device  102  includes a memory  104 , a coprocessor  114 , at least one processor  116 , a communication system  118 , an interface bus  112 , an input/output controller  120 , and one or more actuators  122 . In addition, one or more haptic actuators  126  may be associated with the electronic computing device  102 . For example, a haptic actuator such as the actuator  126  may be embedded in a haptic vest directly associated with the electronic computing device  102 . An interface bus  112  provides power and data communication to the memory  104 , the processor  116 , the coprocessor  114 , the input/output controller  120  (also referred to as I/O  120 ), the communication system  118  and one or more actuators  122 . The I/O controller  120  is connected with other devices such as a display  130 , at least one speaker  124 , at least one actuator  126 , and at least one input device  128  such as a keyboard, a mouse, a gamepad, a joystick, a touch panel, or a microphone. In some embodiments, the one or more actuators  126  may be embedded in one or more input device  128 , for example, a keyboard, a mouse, a gamepad, a joystick, a touch panel, or a microphone. Alternatively, the one or more actuators  126  may be directly interfaced with the electronic computing device  102 . 
     The I/O controller  120  provides power, control information, and enables data communication between the display  130 , the speaker  124 , the actuator  126 , and the input device  128 . Alternatively, the display  130 , the speaker  124 , the actuator  126 , and the input device  128  can self power by a battery or a regulated power supply. In addition, the I/O controller  120  may provide data communication to these devices through a wired or a wireless connection. 
     The memory  104  comprises an operating system  106 , one or more applications  108 , and a haptic module  110 . The haptic module  110  includes computer executable instructions to produce a haptic signal from an audio signal for providing an immersive haptic experience. The haptic module  110  exchanges data and information with other components/devices such as one or more actuators  122  and/or the one or more actuators  126 . Additionally, the haptic module  110  can communicate with the cloud  140 , the server  160 , the distributed system  150  through the communication system  118 . 
     The memory  104  can be a Read-Only Memory (ROM), Random-Access Memory (RAM), digital storage, magnetic tape storage, flash storage, solid-state device storage or some other type of storage device. The memory  104  can store encrypted instructions, source code, binary code, object code, encrypted compiled code, encoded executable code, executable instructions, assembly language code or some other type of computer readable instructions. 
     In some embodiments, the haptic module  110  can be implemented as a separate module having a dedicated processor and memory. For example, the haptic module  110  may be a SoC or implemented in memory  104  associated with a microcontroller. 
     The processor  116  and the coprocessor  118  are enabled to provide hyper-threading, multi-tasking, and multi-processing. Alternatively, the processor  116  can be a special purpose processor or some other type of microprocessor capable of processing analog or digitalized audio signals. The processor  116  and the coprocessor  118  can implement special hardware that is designed for digital signal processing, for example, MMX technology provided by Intel®. MMX technology provides an additional instruction set to manipulate audio, video, and multimedia. The processor  116  can any type of processor such as MMX, SSE, SSE2 (Streaming SIMD Extensions 2), SSE3 (Streaming SIMD Extensions 3), SSSE3 (Supplemental Streaming SIMD Extensions 3), SSE4 (Streaming SIMD Extensions 4) including the variants SSE 4.1 and SSE4.2, AVX (Advanced Vector Extensions), AVX2 (Haswell New Instructions), FMA (Fused multiply—add) including FMA3, SGX (Software Guard Extensions), MPX (Memory Protection Extensions), Enhanced Intel SpeedStep Technology (EIST), Intel® 64, XD bit (an NX bit implementation), Intel® VT-x, Intel® VT-d, Turbo Boost, Hyper-threading, AES-NI, Intel® TSX-NI, Intel® vPro, Intel® TXT, Smart Cache or some other type of implementation for a processor. The processor  116  or the coprocessor  118  can be a soft processor such as the Xilinx MicroBlaze® processor that can include at least one microcontroller, real-time processor, an application processor and the like. 
     The communication system  118  can interface with external devices/applications via wired or wireless communication. For example, the communication system  118  can connect to a server  160  via a wired cable. The communication system  118  has an encoder, a decoder, and provides a standard interface for connecting to a wired and/or wireless network. Examples of communication interfaces include, but are not limited to, Ethernet RJ-45 interface, thin coaxial cable BNC interface and thick coaxial AUI interface, FDDI interface, ATM interface and other network interfaces. 
     The cloud computing environment on the cloud  140  may include computing resources and storage. The storage may include one or more databases with at least one database having information about different actuators, devices in which actuators are embedded or associated, haptic hardware, haptic game specific data, haptic preferences of users, and content information such as gaming information including game type. 
     The server  160  is multi-processor, multi-threaded, with a repository comprising databases, which includes one or more databases having actuator specific information, device specific information, and content information for example computer games including the type of game. The distributed system  160  includes distributed databases that hold information about actuator specific information, device specific information, and content information such as computer games and the different attributes of the games like type, number of players, etc. 
     In some embodiments, the actuator specific information is related to the specification data of the actuator. Similarly, the device specific information may be related to specification data of the electronic computing device  102  in which the actuator is embedded. In some embodiments, the manufacturer of the actuator and the electronic computing device  102  may be different. Therefore, the specification of both the electronic computing device  102  and the actuator are required, even though the actuator is embedded in the electronic computing device  102 . In preferred embodiments, the device specific information includes the device specification along with the actuator specific information, which is embedded in the device. 
       FIG.  2    illustrates different parts of a haptic module in an embodiment. The haptic module  110  includes an audio preprocessor module  202 , an impulse processing module  204 , an audio analysis module  206 , an authoring tool  208 , a transformation module  210 , an aggregation and file management module  212 , a resynthesis module  214  an artificial intelligence processing module  216 , and a database module  220 . 
     In preferred embodiments, the haptic module  110  is stored in the memory  104  of the electronic computing device  102 , which can be a desktop computer, a laptop, a gaming console, a mobile computing device such as a phone or a tablet, a gaming controller such as a joystick, gamepad, flight yoke, gaming mouse, gaming keyboard, keyboard wrist rest, mouse pad, headphones, a virtual computing environment, an electronic gaming composer, a gaming editing application running on a server or a cloud or some other computing device. In some embodiments, the resynthesis module  214  may be implemented separately in different devices, which can process the haptic file to produce an immersive haptic experience. 
     In another variation of the implementation, the resynthesis module  214  includes a synthesizer for generating a haptic output by parsing a computer readable file. The resynthesis module  214  may include one or more actuators connected either directly or through a mixer, which mixes an array of amplitude time values and an array of frequency time values to drive one or more actuators to provide an immersive haptic experience. 
     In some embodiments, the cloud  140 , the server  160 , the distributed system  150  may allow one or more game developers to use authoring tools concurrently, share information, share feedback, and communicate with each other for authoring games. 
       FIG.  3    illustrates the different modules of a haptic module implemented in distributed environments in an embodiment. The haptic module  300  may reside on the cloud  140  or the server  160  or over the distributed system  150 . 
       FIG.  3    shows only one implementation of the haptic module  300  with different modules distributed over the network and residing in different devices, however, there can be other implementations of the haptic module  300  having fewer or more modules residing over a network on different devices. For example, in one implementation, the audio preprocessor module  202 , the impulse processing module  204 , the audio analysis module  206 , the artificial intelligence module  216 , the transformation module  210 , the aggregation and file management module  212 , and the resynthesis module  214  all reside on the cloud  140 . The database module  220  has a processor  318  and associated memory and resides as a distributed database over a network  302 . The electronic computing device  102  includes the authoring tool  208  for analyzing the audio signal and authoring haptic events. 
     Each module has a dedicated processor and memory. In different implementations, different modules may be distributed over the network  302 , For example, the audio preprocessor module  202  has a processor  304 , the impulse processing module  204  has a processor  306 , the audio analysis module  206  has a processor  308 , the artificial intelligence module  216  has a processor  310 , the transformation module  210  has a processor  312 , the aggregation and file management module  212  has a processor  314 , and the resynthesis module  214  has a processor  316 , and the authoring tool  208  can also have a processor, if the authoring tool resides outside the electronic computing device  102 . 
     By way of example and not a limitation, in another variation of this implementation, the audio preprocessor module  202 , the impulse processing module  204 , the audio analysis module  206 , the artificial intelligence module  216 , the transformation module  210 , the aggregation and file management module  212 , the resynthesis module  214 , and the authoring tool  208  reside on the server  160 . The database module  220  can be a distributed database or a network implemented database residing over the network  302 . 
     Other variations and permutations are also possible for deploying different modules on different devices distributed over the network  302 . For example, the audio preprocessor module  202 , the impulse processing module  204 , the audio analysis module  206 , the artificial intelligence module  216 , the transformation module  210 , the aggregation and file management module  212 , the resynthesis module  214 , the authoring tool  208 , and the database module  220  are also possible. 
       FIG.  3    is an exemplary illustration and should not be construed as limiting for the implementation of the haptic module  300  over the network  302 . 
       FIG.  4 A  illustrates different components of an audio analysis module for converting the audio signals into a haptic signal in an embodiment. The haptic processing module  400 A receives the audio signal at the pre-processor module  202 . The preprocessor module  202  removes unwanted frequencies, distortion and other non-linear characteristics from the audio signal. The preprocessed audio signal is passed to the audio analysis module  206  for further processing of the audio signal. The audio analysis module  206  processes the preprocessed audio signal for a fixed time to produce an array of time amplitude values and an array of spectral centroid values. 
     In some embodiments, the audio analysis module  206  processes the preprocessed audio signal for a fixed window. In another embodiment, the audio analysis module  206  processes the preprocessed audio signal for a fixed number of frames. In yet another embodiment, the audio analysis module  206  processes the preprocessed audio signal for a fixed number of audio samples. 
     To produce the array of time amplitude values, the preprocessed audio signal is passed to an envelope approximation and smoothing module  402  to approximate the preprocessed audio signal into time amplitude values, and to smoothen the time amplitude values. The array of time amplitude values is passed to a breakpoint reduction module  404 . The breakpoint reduction module  404  reduces the array of time amplitude values into a linear approximation of a time sequence. Finally, an amplitude envelope module  406  produces an envelope of the linear approximated time sequence and then passes the array of amplitude time values to the authoring tool  208 . In some embodiments, the array of time amplitude values received from the amplitude envelope  406  are for different frequency bands. Each frequency band may comprise an array of time amplitude values. For example, a frequency band ranging from 40 Hz to 100 Hz may comprise the array of time amplitude values for frequencies between 40 Hz and 100 Hz (includes frequencies 40 Hz and 100 Hz). 
     Parallelly, the preprocessed audio signal is passed to the DC offset module  408 . The DC offset module  408  ensures that the preprocessed audio signal always has a non-zero value during an audio silence. This is achieved by adding a small value (like 0.01) to the audio sampled values so that the sampled values are always non-zero, that is, the sampled values always have a positive value. The processed signal is then passed to the spectral centroid module  410 , which calculates the spectral centroid or the center of mass of the received signal. The array of time amplitude values and the array of spectral centroid values are calculated in parallel for the audio signal. In an alternative embodiment, the array of time amplitude values and the spectral centroid values are calculated separately. 
     To calculate the spectral centroid of the received audio signal from the preprocessor module  202 , the preprocessed audio signal is transformed into a frequency domain by performing a Short Time Fourier Transform (STFT) or Fast Fourier Transform (FFT). The spectral centroid of a predefined number of samples is calculated by using a weighted average method. In some embodiments, the number of samples may be fixed. Alternatively, the audio analysis module  206  may automatically determine the number of samples required for calculating the spectral centroid. The average frequency of each frequency interval is calculated; the average frequency is multiplied by the spectral energy of that frequency interval to calculate the spectral energy distribution for that frequency band. Similarly, the spectral energy distribution for all frequency bands is calculated. The sum of energy frequency distribution of all the frequency bands is divided by the sum of the average frequency of all the frequency bands to arrive at the spectral centroid. To calculate the array of spectral centroid values, a fixed number of samples are removed and an equal number of samples are added. For example if the buffer size of the frame of the preprocessed audio signal is  1024  samples, each iteration may replace  128  samples and introduce  128  new samples. In this example, the array of spectral centroid values may be obtained. In some embodiments, the iteration is performed by replacing a fixed number of samples for calculating the time amplitude values and spectral centroid values. 
     In some embodiments, the spectral centroid value may be calculated using a fixed number of audio frames. Furthermore, the array of spectral centroid values may be calculated by replacing a fixed number of audio frames by an equal number of audio frames. For example if the buffer size of the frame of the preprocessed audio signal is  1024  audio frames, each iteration may replace  128  audio frames and introduce  128  new audio frames to be processed. 
     In some embodiments, the spectral centroid value may be calculated using a fixed window size comprising audio data or audio video data. In this implementation, the array of spectral centroid values may be calculated by removing a fixed number of audio data and shifting the window to include an equal number of additional unprocessed audio data. For example if the window size of the preprocessed audio signal is  1024  units, each iteration may replace  128  audio data units and introduce new unprocessed  128  audio units. In some embodiments, the window size is automatically calculated by the audio analysis module  206 . 
     The array of time amplitude values and the array of spectral centroid values are passed to the authoring module  208 . The authoring module  208  includes a user interface that allows the editing and/or modification of the array of time amplitude values and the array of spectral centroid values. 
     In some embodiments, the authoring module  208  is connected with the database  220  having the actuator specific information  222  and the device specific information  224  to allow a user to adjust the array of amplitude time values and the array of spectral centroid values according to electronic computing device  102  having an embedded actuator  122  or an actuator  126 . In one variation of this implementation, the device specific information  224  may include the actuator specific information  222 . For example, when the actuator  122  is embedded within the electronic computing device  102 , the device specific information  224  may include the actuator specific information  222 . 
     The transformation module  210  receives the authored array of time amplitude values and the authored array of spectral centroid values to determine if the authored array of time amplitude values and the authored array of spectral centroid values can fit within a haptic perceptual bandwidth for each of the frequency bands. 
     The combined bandwidth of the actuator and the electronic computing device, where the human can feel vibration, is referred to as “haptic perceptual bandwidth”. The haptic perceptual bandwidth can be calculated from the actuator specific information  222  and device specific information  224  stored in the database module  220 . To illustrate the haptic perceptual bandwidth through an example, let the bandwidth of the actuator  122  be 40 Hz and the bandwidth of the electronic computing device  102  be 120 Hz then if the human can perceive the vibration is 80 Hz then the combined haptic perceptual bandwidth of the electronic computing device  102  having the embedded actuator is 80 Hz. If the authored array of time amplitude values and the authored array of spectral centroid values can fit within a haptic perceptual bandwidth for each of the frequency bands, then the array of time amplitude values and the spectral centroid values is passed to an aggregation and file management module  212  for creating a computer readable haptic file. The computer readable haptic file can be parsed by the actuator  126 . 
     Otherwise, the authored array of time amplitude values and the authored array of spectral centroid values are transformed into a transformed array of time amplitude values and a transformed array of the spectral centroid values by implementing an algorithm. The algorithm performs the steps of determining the rank of each frequency band, if the frequency band ranks are not already provided. If the frequency bands are not ranked, then the algorithm calculates the rank of each frequency band. Subsequently, the algorithm tries to fit the transformed array of time amplitude values and the transformed array of the spectral centroid values based on the ranking of the frequency bands. When only one frequency band can fit into the haptic perceptual bandwidth, the algorithm performs the processes of shifting of frequency bands. To perform the process of shifting the frequency bands, the algorithm determines the resonant frequency of the electronic computing devices  102  (along with the actuator  122 ) and calculates the distance between the determined resonant frequency and the nearest frequency band. The positive value of the calculated distance is always used. The nearest frequency band is shifted to the resonant frequency of the electronic computing device  102 . The algorithm then shifts all the other frequency bands so that they lie close to the resonant frequency of the electronic computing device  102 . The algorithm then determines if all the frequency bands can be accommodated within the haptic perceptual bandwidth. If all the frequency bands can be accommodated or fit into the haptic perceptual bandwidth then the algorithm fits all the frequency bands into the haptic perceptual bandwidth. Otherwise, the algorithm fits the frequency bands based on the rank of each frequency band. The frequency bands are accommodated based on descending order of ranks, that is, the highest ones are accommodated before the lower ranked frequency bands. For example, the frequency band f 1  is ranked  2 , frequency f 3  is ranked  1 , frequency band f 2 , is ranked  3  then the frequency bands are accommodated in the order f 3 , f 1  and f 2 . If all the frequency bands cannot be accommodated then the frequency bands that cannot be accommodated into the haptic perceptual bandwidth are discarded. For example, if the frequency band f 2  cannot be accumulated into the haptic perceptual bandwidth then the frequency band f 2  is discarded. 
     If no frequency ranks are provided, then the algorithm ranks the frequency bands before shifting the frequency bands. When only one frequency band, which is nearest to the resonant frequency, can be accommodated within the haptic perpetual bandwidth, then the algorithm shifts that frequency band and eliminates all the other frequency bands. 
     The output of the algorithm is the transformed audio descriptor data comprising a transformed array of time amplitude values and a transformed array of spectral centroid values, which is provided to an aggregation and file management module  212 . The aggregation and file management module  212  converts the transformed audio descriptor data into a computer readable haptic file. 
     Subsequently, the output of the aggregation and file management module  212  is saved as a computer readable haptic file, which can be parsed and processed by the resynthesis module  214 . The resynthesis module  214  processes the transformed audio descriptor data stored in the computer readable haptic file to produce haptic experience in one or more actuators  122  or one or more actuators  126 . 
     In one variation of this implementation, the array of time amplitude values control the amplitude of the actuator whereas the array of spectral centroid values provide the corresponding frequency to be set in one or more actuators for providing an immersive haptic experience. 
       FIG.  4 B  illustrates the various components of an audio analysis module implementing a filter bank for converting an audio signal into a haptic output in another embodiment. The audio analysis module  206  receives the preprocessed audio signal from the audio preprocessor module  202 . The audio analysis module  206  comprises a filter bank  440 , an envelope follower  450 , an envelope smoother  460 , and a data reduction  470 . 
     The audio signal received from the audio preprocessor module  202  is separated into different frequency bands by the filter bank  440 . Each frequency band has a center frequency and bandwidth. In some embodiments, the center frequency may be a median value or the modal value of the frequencies of the frequency band. In some embodiments, the centre frequency and the bandwidth of each frequency band can be selected by a user through a Graphical User Interface (GUI) of the authoring tool  208 . 
     The filter bank  440  includes a band filter  422 , a band filter  424 , a band filter  426 , and a band filter  428 . Each band filter is tuned to pass a predefined band of frequencies. Although only four band filters are shown in the filter bank  440 , in other variations the filter bank  440  may include more than or less number of band filters. Each filter band may be tuned to different frequencies. For example, the band filter  422  may be tuned to pass 0-60 Hz. Likewise, the band filter  424  may be tuned to pass frequencies tuned at 60-120 Hz. Similarly, the band filter  426  may be tuned for frequencies ranging from  120 - 180  Hz. In some embodiments, each band filter can be tuned at an unequal frequency range. For example, the band filter  422  is tuned for frequencies ranging from 20 to 60 Hz; the band filter  424  is tuned to band frequencies ranging from 60-120 Hz; the band filter  426  may be tuned to frequencies ranging from 120 to 200 Hz; the band filter  428  may be tuned to frequencies ranging from 200 Hz to 1 kHz. The output of the filter bank  440  is a plurality of filtered audio signals, each signal belonging to one frequency band. For example, each band filter such as the band filter  422 , the band filter  424 , the band filter  426 , and the band filter  428  produce an audio signal as per the tuned frequency band of the band filter. 
     The output of the filter bank  440  is provided to the envelope follower  450 . The envelope follower  450  includes a band envelope approximation  452 , a band envelope approximation  454 , a band envelope approximation  456 , and a band envelope approximation  458 . In the current implementation, the band envelope approximation  452  receives the filtered audio signal from the band filter  422 . Likewise, the band envelope approximation  454 , the band envelope approximation  456 , and the band envelope approximation  458  receive the filtered audio signal from the band filter  424 , the band filter  426 , and the band filter  428  respectively. The center frequency of each band filter can be evenly spaced over a frequency range in a linear or logarithmic scale. 
     The envelope follower  460  generates a time-amplitude envelope for each frequency band. The centre frequency for each frequency band is included in the time amplitude envelope. The envelope follower  460  includes a band envelope approximation module  452 , a band envelope approximation module  454 , a band envelope approximation module  456 , and a band envelope approximation module  458 . Each band approximation module implements one or more envelope approximation algorithms. Each band approximation module such as band approximation module  452  can have at least one envelope follower, a memory bank, and an optional processor. Furthermore, in the current implementation, the envelope follower  450  is utilized for generating time amplitude envelope for one or more frequency band using the band envelope approximation such as band envelope approximation  458 , however, in other implementations, the envelope follower  450  may implement other type of envelope approximation methods, for example, a Hilbert transformation. For each frequency band, the time amplitude envelope is approximated into an array of time amplitude values/data points that best represents the amplitude values over time for each frequency band. 
     The output of the envelope follower  450  is passed to the envelope smoother  460 . The envelope smoother  460  includes a band envelope smoother  462 , a band envelope smoother  464 , a band envelope smoother  466 , and band envelope smoother  468 . In the current implementation, the band envelope smoother  462  receives the time-amplitude envelope from the band envelope approximation  452 . Likewise, the band envelope smoother  464 , the band envelope smoother  466 , and the band envelope smoother  468  receive the time-amplitude envelope from the band envelope approximation  454 , the band envelope approximation  456 , and the band envelope approximation  458  respectively. 
     The band envelope smoother  460  smoothens the time amplitude envelope to reduce abrupt signal changes in order to generate a smooth time amplitude envelope at the center frequency for each of the frequency bands. Due to large variations in the amplitude values, there can be abrupt signal changes; these abrupt signal changes are smoothened using the envelope smoother  460 . The smoothing process eliminates outliers, clips of sharp peaks, and produces a smoothed time-amplitude envelope for each frequency band. The band envelope smoother  460  has multiple band smoothers, one for each of the frequency bands. Each band envelope smoother such as the band envelope smoother  462  includes at least one digital filter, a memory bank, and an optional processor. Each band envelope smoother such as the band envelope smoothers  462  can be a digital filter, for example, the digital filter can be a low-pass Butterworth filter with a cut-off frequency of 250 Hz. However, in other implementations, the band envelope smoother  462  may include different types of digital filters, which may be set to different cut-off values ranging between 30 Hz to 1000 Hz. 
     The output of the envelope smoother  460  is passed to the data reduction  470 . the band data. The data reduction  470  includes a band data reduction  472 , a band data reduction  474 , a band data reduction  476 , and a band data reduction  478 . The band data reduction  472  receives the smoothed time amplitude envelope from the band envelope smoother  460 . Likewise, the band data reduction  474 , the band data reduction  476 , and the band data reduction  478  receive the smoothed time-amplitude envelope from the band envelope smoother  464 , the band envelope smoother  466 , and the band envelope smoother  468 . 
     The data reduction  470  reduces the number of time amplitude data points or the time amplitude values of the smoothened time-amplitude envelope to produce a reduced time amplitude band envelope. For each band data reduction such as band data reduction  472  the produced reduced time amplitude values. The array of reduced time amplitude band envelopes includes the center frequency and array of reduced time amplitude values/data points for each frequency band. The data reduction  470  may include a memory bank and an optional processor. The data reduction  470  reduces the smoothed time amplitude envelope into a minimum number of time amplitude values/data points. The reduced time amplitude values are generated without losing information or losing minimal information. Finally, the audio analysis module  206  produces an analyzed array of time amplitude values, the centre frequency, and an array of time frequency values for each frequency band. 
     For each frequency band, the audio analysis module  206  may process the received audio signal to produce an array of time amplitude values, a centre frequency and an array of spectral centroid values as described in  FIG.  4 A . Each frequency band may be analyzed separately and passed to the authoring tool  208  for authoring the array of time amplitude values, the centre frequency and the array of spectral centroid values for each frequency band. Thereafter, the authored array of time amplitude values, the authored centre frequency and the authored array of spectral centroid values for each frequency band is provided to the transformation module  208  for generating the transformed audio descriptor data. 
     In one implementation, the data reduction  470  utilizes the Ramer-Douglas-Peucker data reduction algorithm in order to minimize the amount of time-amplitude data points to a manageable proportion. In different implementations, the data reduction algorithms can include piecewise linear approximation methods such as, but not limited to, RLS (recursive least square), Visvalingam-Wyatt, differential evolution, Broyden-Fletcher—Goldfarb-Shanno (BFGS), gradient descent and other known techniques. 
     In a different implementation, the audio analysis module  206  may include an audio analysis processor, a digital signal processor, and a memory to store and execute envelope smoothing algorithms and techniques, for example, numerical analysis, B -splines, AI algorithms and other known techniques. 
       FIG.  5 A  illustrates the block diagram of an audio analysis module implementing a harmonic-percussive source separation for converting an audio signal into a haptic output in an embodiment. The audio analysis module  206  receives the preprocessed audio signal from the preprocessed module  202  and passes it to a HPSS module  502  to produce a power spectrogram. In one variation of this implementation, the HPSS module  502  resides in the audio analysis module  206 . In another variation of this implementation, the HPSS module  502  is a separate module associated with the audio analysis module  206 . The HPSS module  502  performs the Short Time Fourier Transform (STFT) or Fast Fourier Transform (FFT) of the received audio signal from the received preprocessor module  202 . The power spectrogram from the HPSS module  502  is passed to a harmonic module  506 , a percussive module  504 , and a residual module  508 . 
     The percussive module  504  receives the power spectrogram from the HPSS module  502 . The percussive module  504  filters the power spectrogram to produce a percussive spectrogram. Simultaneously, the harmonic module  506  receives the power spectrogram and filters it to produce a harmonic spectrogram. The residual module  508  calculates the residual spectrogram by adding the harmonic spectrogram and the percussive spectrogram and then subtracting the sum of the harmonic spectrogram and the percussive spectrogram from the power spectrogram. The power spectrogram may be filtered horizontally and/or vertically to produce the harmonic spectrogram and/or the percussive spectrogram. 
     The percussive spectrogram from the percussive module  504  is passed to a centroid tracker  510  to calculate the spectral centroid of the percussive spectrogram. The spectral centroid may be calculated for a fixed frame of audio data. Alternatively, the spectral centroid may be calculated for fixed window size of the audio data packets or variable window size of the audio data packets. After calculating the spectral centroid of the percussive spectrogram, the spectral centroid values are passed to an envelope approximation and smoothing module  512 . The envelope approximation and smoothing module  512  perform the function of (a) approximate the preprocessed audio signal into an time amplitude frequency values, and (b) smoothen the approximated time amplitude frequency values. The output is the array of time amplitude frequency values, which is then passed to a breakpoint reduction module  514 . The breakpoint reduction module  514  reduces the array of time amplitude frequency values based on a linear approximation of a time sequence to produce an array of time amplitude frequency values with best fit approximation and minimum number of data points or data values without losing any information in the audio signal or minimizing the loss of information in the audio signal. Finally, the array of time amplitude values are provided to an amplitude envelope module  516  for extracting the time amplitude envelope comprising an array of time amplitude values. 
     Simultaneously, the percussive spectrogram from the percussive module  502  is passed to the transient module  518 . The transient module  518  detects the presence of transients in the percussive spectrogram. The transient processing is discussed in detail in the US application Ser. No. 16/435,341 (which is incorporated here by reference). When transients are detected in the percussive spectrogram, the transient module  518  passes the transients to an impulse sequence module  520 , which produces a sequence of impulses. The impulse sequence comprises an array of time frequency values. In one variation of this implementation, the impulse sequence may comprise an array of time frequency values, an array of time amplitude values and/or array of time amplitude frequency values. 
     The harmonic module  506  produces the harmonic spectrogram, which is passed simultaneously to an envelope approximation and smoothing module  522  and to a centroid tracker  528 . The harmonic spectrogram is passed to the envelope approximation and smoothing module  522 , which produces an array of amplitude frequency time values. The envelope approximation and smoothing module  522  perform the function of (a) approximate the preprocessed audio signal into an time amplitude frequency values, and (b) smoothen the approximated time amplitude frequency values. The array of time amplitude frequency values is then passed to a breakpoint reduction module  524 . The breakpoint reduction module  524  reduces the array of time amplitude frequency values based on a linear approximation of a time sequence. Finally, the array of time amplitude values are provided to an amplitude envelope module  526  for extracting the time amplitude envelope having an array of time amplitude values. 
     The harmonic spectrogram, in parallel, is provided to the centroid tracker  528  to calculate an array of spectral centroid values of the harmonic spectrogram. . The array of spectral centroid values is passed to an envelope approximation and smoothing module  530  to (a) approximate the preprocessed audio signal into an array of time frequency values, and (b) smoothen the approximated array of spectral centroid values. The array of spectral centroid values is then passed to a breakpoint reduction module  532 . The breakpoint reduction module  532  reduces an array of spectral centroid values based on a linear approximation of a time sequence. Finally, the array of spectral centroid values are passed to a frequency envelope module  534  for providing a frequency envelope comprising an array of time frequency values. 
     The residual module  508  calculates the residual spectrogram from the power spectrogram and passes the residual spectrogram to a spectral envelope approximation module  536 . The spectral envelope approximation module  536  performs (a) approximates the residual spectrogram into an array of time amplitude frequency values, and (b) smoothen the time amplitude frequency values. The array of time amplitude frequency values is then passed to the breakpoint reduction module  538 . The breakpoint reduction module  538  reduces the array of time amplitude frequency values based on a linear approximation of a time sequence. Finally, the array of time amplitude frequency values are passed to a spectral envelope module  540  for extracting a residual array of time amplitude frequency values. 
     In some embodiments, the envelope approximation and smoothing module  512  calculates the array of time frequency values for a fixed number of samples or fixed time. 
     In some embodiments, the array of spectral centroid values is calculated for a predefined number of samples over a fixed time or for a fixed number of samples, that is, a fixed window size. 
     To calculate the array of spectral centroid values, the Fast Fourier Transform or Short Time Fourier Transform is taken for a fixed number of samples to generate a spectrum of spectral energy vs. frequency. For each frequency band, the average frequency is calculated; the average frequency of each frequency band is multiplied by the spectral energy of that frequency band. The sum of the product of average frequency and spectral energy of all frequency bands is calculated, which is divided by the sum of spectral energies of all frequency bands. After calculating the spectral centroid, the samples are left shifted, that is, removing the predefined number of samples and introducing the same number of new samples. Subsequently, the spectral centroid is calculated for these sampled values. This results in an array of spectral centroid values produced, which are provided to the authoring module  208  for authoring a percussive stream, a harmonic stream, and a residual stream. 
     In some embodiments, the authoring module  208  may allow the editing, addition, deletion or modification of the array of spectral centroid values. 
       FIG.  5 B  illustrates the block diagram of an audio analysis module implementing a harmonic-percussive source separation for converting an audio signal into a haptic output in another embodiment. The haptic processing system  500 B includes an additional impulse processing module  204  in this alternate implementation. The array of time amplitude values from the amplitude envelope module  516  and the array of impulse sequences from the impulse sequence module  520  processed from the percussive spectrogram. The array of time amplitude values from the amplitude envelope module  526 and the array of time frequency values from the amplitude envelope  534  processed from the harmonic spectrogram are provided to an impulse processing module  204 . The impulse processing module  204  implements algorithms for the amplitude envelope module  516  and the amplitude envelope module  526  to estimate the occurrence of an impulse (i.e. transient/emphasis) for any given breakpoint in each amplitude envelope received from the amplitude envelope module  516  and the amplitude envelope module  526 . A comparator then compares the impulse sequence  520  with the resulting impulse sequences derived from the amplitude envelope module  516  and the amplitude envelope module  526  to add or remove impulses. The impulse processing module  204  can make changes to the timing of each impulse, so that the transients are consistent and aligned with the audio signal. Furthermore, the frequency information from the frequency envelope  534  is utilized to set the sharpness (frequency) value for each impulse returned by the impulse processing module  204 . 
     In addition, the impulse processing module  204  monitors the ducking to the amplitude envelope, This allows the emphasis on the impulses during haptic synthesis providing an immersive haptic experience. The output of the impulse processing module  204  is provided to the authoring tool  208  and subsequently to the transformation module  210 . Finally, the transformed array of time amplitude values, the transformed array of impulse sequences, and the transformed array of time frequency values are provided to the aggregation and file management module  212  to create a computer readable haptic file. 
       FIG.  5 C  illustrates the block diagram of the impulse processing module in an embodiment. The impulse processing module  204  comprises a comparator module  570 , an impulse amplitude algorithm module  572 , a sharpness and onset module  574  and an amplitude ducking module  578 . In some embodiments, different modules of the impulse processing module  204  may have a processor and an associated memory. 
     The comparator module  570  receives the array of centroid frequency values from the harmonic spectrogram and an array of impulse sequences from the percussive spectrogram. In addition, the comparator module  570  also receives an input from the sharpness and onset module  574 . The impulse amplitude algorithm module  572  receives the array of time amplitude values from the harmonic spectrogram, processes the array of time amplitude values and passes the same to the sharpness and onset module  574 . The array of time amplitude values from the harmonic spectrogram is also passed to the amplitude ducking module  578 . 
     In some embodiments, the amplitude ducking may be performed continuously at least a few milliseconds before the actual impulse sequence. In different embodiments, the time duration may vary from 0.5 ms to 100 ms. In the preferred embodiment, the amplitude ducking may be performed at 0.5 ms to 15 ms before the arrival of the impulse sequence. 
     In some embodiments, the amplitude ducking may be performed based on deep learning algorithms or other artificial intelligence algorithms. The timing and duration of the amplitude ducking is determined by deep learning algorithms, which have been previously trained using test data. 
     In some embodiments, the amplitude ducking may be performed based at least on one of the device specific information, the actuator specific information and game context specific information. 
     In some embodiments, the amplitude ducking may be performed based on the previous data corresponding to the array of time amplitude and the array of time frequency values. 
     In some embodiments, the amplitude ducking may be implemented with a time delay. The delay may be of fixed time or a variable time. In some embodiments, the time delay may be implemented using deep learning algorithms. Additionally, the machine learning algorithms may implement the delay that may be different for different impulse sequences. For example, the delay of  1  ms for the first set of impulses and  5  ms for the second set of impulses. 
     In some embodiments, the amplitude ducking may be controlled by the comparator based on the look ahead algorithm. The look ahead information received by implementing the look ahead algorithm may be based at least on the audio data such as amplitude, frequency and phase. 
     The comparator  570  also provides feedback to the amplitude ducking module  578 . The purpose of providing the feedback from the comparator  570  to the amplitude ducking  578  is to ensure that the impulse signals generated by both the percussive audio stream and the harmonic audio stream do not interfere with each other. The continuous audio stream is generated by the array of time amplitude values corresponding to the harmonic spectrogram. Thus, whenever the impulse or sequence of impulses are detected by the comparator  570 , a feedback is passed to the amplitude ducking  578 , which suppresses the continuous audio stream of the harmonic spectrogram until the impulse/ impulse sequence has passed. This ensures a good haptic experience. The output of the comparator  570  is an array of time frequency values and an array of time amplitude values for impulse generation. 
     In some embodiments, the amplitude ducking  578  may partially suppress the amplitude of the continuous signal, for example, the amplitude of the continuous stream is reduced by a fixed percentage, for example 25%. In another implementation, the amplitude of the continuous stream is reduced by 100%. In other implementations the amplitude of the continuous stream is reduced between 50% -90%. 
     The impulse processing module  204  utilizes both the harmonic spectrogram and the percussive spectrogram to generate impulses. 
     In some embodiments, the array of time frequency values and the array of time amplitude values for impulse generation is passed to an impulse actuator for producing the haptic effect. In embodiments, the impulse actuator may be an LRA, an ERM, a wideband actuator, a piezo actuator or some other type of actuator. 
     In some embodiments, at least two different actuators are connected to play back the haptic effect. The first actuator that produces the impulse effect is connected to the array of frequency time values and the amplitude of the first actuator is controlled by the array of the time amplitude values, which is obtained from both the percussive spectrogram and the harmonic spectrogram. The second actuator is controlled by the array of time frequency values and the array of amplitude time values to produce a continuous haptic effect. 
     In some embodiments, the continuous haptic effect produced by the second actuator may be ducked to accommodate the impulse haptic effect produced by the first actuator. The amplitude ducking provides a better haptic experience. 
       FIG.  5 D  illustrates the block diagram of an authoring tool for processing audio data in an embodiment. The authoring tool  208  comprises an impulse editor  580 , an analysis parameters user interface  582 , a continuous editor  584  and a residual editor  588  apart from other interface tools. The impulse editor  580  allows the user to modify and/or edit the array of time frequency values of the frequency envelope and the array of impulse sequence values for the percussive spectrogram. The output of the impulse editor  580  is an authored frequency envelope comprising the array of time frequency values and an authored array of impulse sequences comprising the array of time amplitude values. 
     The analysis parameters user interface  582  allows the user to adjust the one or more parameter values received from the impulse processing system  204 . The impulse values may be calculated by the gradient at specific signal values. The change in the gradient is greater than a predetermined value a sequence of impulses emerge. The user may modify and/or adjust these impulses. For example, the user may modify the amplitude of each impulse or eliminate one or more impulses. The analysis parameters user interface  582  also allows the user to edit/change/modify or adjust the parameters in the impulse amplitude algorithm  572  and the amplitude ducking  578 . 
     The array of time amplitude values from the amplitude envelope module of the harmonic module  506  and the array of time frequency values from the frequency envelope associated with harmonic module  506  can be edited or adjusted using a continuous editor  584 . Likewise, the array of time amplitude frequency values from the residual spectrogram  508  can be edited or modified by a residual editor  586 . For example, the user may edit the array of time amplitude values from the amplitude envelope module of the harmonic module  506  and the array of time frequency values from the frequency envelope associated with harmonic module  506  to adjust the haptic feedback. 
     In some embodiments, the user may create two sets of authored array of time amplitude values and the authored array of time frequency values one for normal power mode and another set of values for low power mode. In this implementation, the amplitudes of the amplitude envelope comprising the array of time amplitude values and the frequency envelope comprising the array of time frequency values may be modified to reduce the power consumption in the low power mode. For example, the amplitude may be scaled by 25 percent and an amplitude threshold is provided. The haptic feedback may occur only when the amplitude is above the amplitude threshold value. 
     The output from the authoring tool  208  comprises the authored array of time frequency values and the authored array of time amplitude values of the continuous stream edited using the continuous editor  584 ; the authored array of impulse sequence having array of amplitude time values and the authored array of time frequency values from the impulse editor  582  and optionally the authored array of amplitude frequency time values from the residual editor  586 . 
       FIG.  6    illustrates a block diagram for processing an authored array of time frequency and an authored array of time frequency values in an embodiment. The authored array of the continuous stream, the authored array of the impulse stream, and the authored array of the residual stream are provided to different playback controllers for generating haptic experience. 
     The output from the impulse editor  580  comprising the authored frequency envelope having the authored array of time frequency values and the authored array of impulse sequence having the authored array of time amplitude values received from the percussive spectrogram is passed to an impulse playback controller  602 . The impulse playback controller  602  receives the frequency envelope and the authored array of the impulse sequence, analyzes it, and extracts the impulse sequence. In addition, the frequency of each impulse is also determined. The extracted impulses and their corresponding frequencies are passed on to an impulse generator  604 . The impulse generator  604  generates impulse signals based on the array of frequency values. The impulse playback controller  602  provides feedback to a gain controller  592 . The output from the impulse generator  604  is provided to the gain controller  608 . The gain controller  608  also receives a feedback signal from the impulse playback controller  602  to adjust the gain of the impulse signals. The output of the gain controller  608  is provided to a mixer  610 . 
     Likewise, the output of the continuous editor  584  is provided to an oscillator playback controller  612 . The oscillator playback controller  612  receives the amplitude envelope comprising the authored array of time amplitude values and a frequency envelope comprising an authored array of time frequency values. The oscillator playback controller  612  receives the amplitude envelope and the frequency envelope to generate a haptic signal, which is passed to an oscillator  614 . The oscillator  614  generates a continuous haptic signal at frequencies and amplitudes received from the oscillator playback controller  612 . The output from the oscillator playback controller  612  is provided to a gain controller  618  as feedback. The gain controller  618  adjusts the gain, that is, the amplitude of the haptic signal based on the feedback provided by the oscillator playback controller  612 . The output of the gain controller  618  is passed to the mixer  610 . 
     In one variation of the implementation, the authored array of time amplitude frequency values received from the residual editor  586  is processed by a residual playback controller  620 . The residual playback controller  620  receives the authored array of time amplitude frequency time values and extracts time amplitude frequency values. The time amplitude frequency values are passed to a filter  622 , which filters the received time amplitude frequency values based on preset filter parameters. The output of the filter  622  is provided to a gain controller  624 . The gain controller  624  also receives a feedback signal from the residual playback controller  620  and accordingly adjusts the gain or the amplitude based on the array of time amplitude frequency values. The inclusion of processing of the residual signal is optional and may be implemented based on predetermined criteria. For example, a substantial presence of a noise component in the authored signal. 
       FIG.  7    illustrates a block diagram for processing an authored array of time frequency and an authored array of time frequency values in another embodiment. In this implementation at least two actuators are used. The authored frequency envelope and the authored impulse sequence are passed to the impulse playback controller  602  and the authored frequency envelope and the authored amplitude envelope are passed to the oscillator playback controller  612 . The output of the impulse playback controller  602  is passed to the impulse generator  604  and to the gain controller  608 . Likewise, the output of the oscillator playback controller  612  to the oscillator  614  and finally to the gain controller  618 . In this implementation, the output from the gain controller  608  is passed to a mixer  702  and the output from the gain controller  618  is provided to a mixer  704 . Feedback from the impulse playback controller  602  is provided to the gain controller  608 . Similarly, feedback from the oscillator playback controller  612  is provided to the gain controller  618 . 
     The mixer  702  and the mixer  704  are controlled by a mixer controller  708 . The mixer controller  708  controls the haptic effect produced by the two actuators  126 . In one implementation, the two actuators  126  may have similar specifications. In another implementation, the two actuators  126  may have different specifications. For example, at least one actuator  126  may be an LRA and another actuator  126  may be a voice coil. 
     In this variation, the impulse playback controller  602 , the impulse generator  604  and the gain controller  608  drive the first actuator  126  through the mixer  702 . The actuator  126  receives a feedback signal from the mixer  704  that is associated with the oscillator playback controller  612 , the oscillator  614 , the gain controller  618 . In addition, the first actuator  126  is controlled by the mixer controller  708  that controls the amount of haptic feedback to be provided to each of the actuators  126 . Likewise, the oscillator playback controller  612 , the oscillator  614 , the gain controller  618  drive the second actuator  126  through the mixer  704 . The second actuator  126  also receives feedback from the mixer  702  associated with the impulse playback controller  602 , the impulse generator  604  and the gain controller  608 . Additionally, the second actuator  126  is controlled by the mixer controller  708  that controls the amount of haptic feedback to be provided to each of the actuators  126 . The mixer controller  708  provides a balance between the haptic effect produced by the impulse signal and the continuous signal. It should be noted that the first actuator  126  and the second actuator  126  in combination provide an immersive haptic experience by processing the impulse signal in the first actuator  126  and the continuous signal played in the second actuator  126 . Other combinations with more than two actuators  126  are possible in this embodiment. For example, the two or more actuators  126  can be attached to the mixer  702 . Likewise, two or more actuators  126  can be attached to the mixer  704 . 
     In some embodiments, the mixer  702  and the mixer  704  are controlled by a mixer controller  708 . The mixer controller  708  may adjust the ratio of the impulse signal and the continuous signal to control the functioning of the two actuators  126 . 
     In another variation of this implementation, the mixer controller  708  may receives input from the device specific information  224 , the actuator specific information  222 , and the content specific information  226  for adjusting the ratio of impulse signal and the continuous signal for optimal performance for a combination of the actuators  122  or the actuators  126  associated with the electronic computing device  102 . 
     In some embodiments, the adjustment may also happen dynamically based on the content specific information  226 . In some other embodiments, the mixer controller  708  may implement machine learning and analytics to predict the optimum ratios of the impulse signal and the continuous signal to control the functioning of the two actuators  126 . 
     In some embodiments, the mixer  702  receives the output from the gain controller  608  and the mixer  704  receives the output from the gain controller  618 . The output from the mixer  702  and the mixer  704  is controlled by the mixer controller  708 . In some embodiments, the mixer controller  708  implements deep learning algorithms for dynamically controlling the mixing of the impulse signal and the continuous signal. 
     In some embodiments, the output of the gain controller  608  is provided to the mixer  702  and simultaneously to the mixer  704 . Similarly, the output of the gain controller  618  is provided to the mixer  702  and to the mixer  704 . In addition, the output of the gain controller  608  and the gain controller  618  is also provided to the mixer controller  708 , which controls the mixing of the signals, that is, the impulse signal (transients) and the continuous signal in appropriate proportion for immersive haptic experience. In one variation of this implementation, the mixer controller  708  may include deep learning algorithms that implement learning algorithms to control the mixing of the signals from the gain controller  608  and the gain controller  618 . 
     In some embodiments, the mixer controller  708  may be associated with an analytics AI, machine learning and may apply a mix ratio of the continuous haptic stream and the impulse haptic stream depending on the incoming content. For example, quiet content has a 70/30 ratio and louder, more dynamic content has a 50/50 ratio. 
     In one variation of this embodiment, a transformed array of the continuous stream, a transformed array of the impulse stream, and a transformed array of the residual stream are provided to different playback controllers for generating haptic experience. In this implementation, the processing of the continuous stream and the impulse stream may happen exactly as disclosed in  FIG.  6    and  FIG.  7   . Furthermore, the mixer  702 , the mixer  704  and the mixer controller  708  may be configured in similar configuration and may operate as described in  FIG.  6    or  FIG.  7   . 
       FIG.  8 A  illustrates the detection of the impulses in the impulse processing module using a gradient approach in an embodiment. The impulse processing module  204  calculates the rate of change of the audio signal to calculate the gradient or the slope for the percussive stream. The slope or the gradient is calculated at each point or at fixed intervals of time, for example, 100 microseconds. If the slope or the gradient is greater than a threshold value, then an impulse signal is generated. The threshold value is a slope greater than 60 degrees and less than 90 degrees. For example, if the gradient is greater than a 60 degree then the impulse processing module  204  may generate an impulse. Likewise, if the gradient is less than 60 degree angle then no impulse is generated as shown in  FIG.  8 A . 
     In addition, the gradient and duration of the impulse may be used to calculate the sharpness. The calculated sharpness value is provided to the comparator  570  to generate the array of time frequency values. In one variation of this implementation, the sharpness may be utilised for merging of the impulses obtained from the percussive spectrogram and the harmonic spectrogram. When the slope or the gradient lies between 50 degrees to 90 degrees, an impulse signal is generated. 
       FIG.  8 B  illustrates the detection of the impulses in the HPSS spectrogram using a gradient approach in another embodiment. The impulse may be determined separately for the harmonic spectrogram and the percussive spectrogram as shown in  FIG.  8 B . The gradient at different points of the harmonic spectrogram and the percussive spectrogram is calculated. When the impulses are detected either in the harmonic spectrogram or the percussive spectrogram, they are passed to the authoring tool  208 . The harmonic stream  810  of the harmonic spectrogram and the percussive stream  820  of the percussive spectrogram are authored and a merged stream  830  of the impulses is produced. In one implementation, the impulses detected in the harmonic stream  810  and the percussive stream  820  are merged by a user through the authoring tool  208 . Alternatively, the merging of the harmonic stream  810  and the percussive stream  820  is performed in real time by a based on a rule-based engine. 
     In another variation of this implementation, the merging of the harmonic stream  810  and the percussive stream  820  into a merged stream  830  is performed automatically using artificial intelligence processing module  216 . 
     In yet another variation of this implementation, deep learning algorithms implemented in the artificial intelligence module  216  are utilized for merging the harmonic stream  810  and the percussive stream  820  into a merged stream  830 . A training dataset of the harmonic stream  810  and the percussive stream  820  is provided for training the deep learning algorithms implemented in the artificial intelligence module  216 . 
     In some embodiments, the gradient value is also used to calculate the sharpness of the signal from the impulse characteristics. The duration of impulse is used to determine the value of the sharpness. The sharpness value is used by the impulse processing module  204  for editing and/or merging the impulses. 
     In another embodiment , the harmonic stream  810  and the percussive stream  820  are processed separately in the impulse processing module  204  and then merged automatically using predefined algorithms. In some embodiments ,the process of merging the harmonic stream  810  and the percussive stream  820  involves suggesting the user through the user interface in the authoring tool  208  and providing the merged stream  830  to the user for editing/modification. 
       FIG.  9    illustrates a Graphical User Interface (GUI) of the authoring tool in an embodiment. The authoring tool  208  provides an exemplary GUI  902 . The GUI  902  displays the audio preprocessed signal  910  as an audio waveform plot. The GUI  902  displays several signal curve editors for the array of time frequency values for each frequency band for the filter bank implementation of  FIG.  4 B . 
     Referring to GUI  902 , a curve editor for a high frequency envelope  912 , a curve editor for mid frequency envelopes  914  and  916 , and a curve editor for a low frequency band envelope  918  is provided. Although, in this implementation only one high frequency band, two mid frequency bands, and one low frequency band are illustrated, in other implementations, there can be multiple frequency bands within the high frequency bands, the mid frequency bands, and the one low frequency band. The curve editors  912 - 918  display the time-amplitude envelopes having editable point for each time amplitude data values/point, which can be dragged by a mouse to either stretch or compress a time amplitude values/data point from the current time amplitude value to a new time amplitude value to change the characteristics of the haptic response. Since the time amplitude values/data points are already reduced in the breakpoint reduction module  470  or the breakpoint reduction module  514  and the breakpoint reduction module  524 , therefore the curve editor allows easy manipulation of time amplitude values/data points. Additionally, time amplitude values/data points can be added or deleted to allow for further customization of the time amplitude envelope. 
     In addition, the GUI  902  also displays an impulse curve editor  920  and a residual spectrograph  922 . The impulse curve editor  920  displays an impulse curve, which is provided by the impulse processing module  204 . The impulse curve editor  920  allows editing and/or modification of the impulse sequence. In addition, the user can manipulate the merged stream  830  using the impulse curve editor  920 . Additionally, the impulse curve editor  920  provides allows a user to drag the impulse curve with a mouse to reshape it. 
     The residual spectrogram comprises a residual noise editor  922  to adjust the noise component. Several noise shaping options are provided in the user interface through the noise type options  936  The noise type options  936  has selectable radio buttons, each radio button providing a specific type of noise shaping. 
     The GUI  902  also includes a number of combo boxes. Only three combo boxes are shown; however in other implementations there may be additional GUI components such as combo boxes or a dropdown box. In this implementation, a combo box  904  is used for selecting the actuator. Each actuator, such as actuators  122  or  126 , has a unique actuator ID. The combo box  904  allows the user to select a specific actuator ID from the list of different types of actuators. Likewise, the combo box  906  allows the user to select the device ID associated with the actuators such as the actuator  122  or the actuator  126 . In some embodiments, when the actuator  122  is embedded within the electronic computing device  102  the user may select a particular device ID, which would automatically fill the actuator ID president in the electronic computing device  102 . The user may select the electronic computing device  102  from a list of devices such as a tablet, a joystick, a gamepad or a mobile phone through a selection of a radio button. Similarly, when a specific actuator ID is selected from the actuator combo box  904  then the device ID displays a list of devices that have the specific actuator embedded within it. In another implementation the contents of the actuator combo box  904  may be populated by querying the database  220 . The combo box  908  is utilized for selecting the game type through a radio button. For example, the game type is divided based on the age and the content of the game such as fight game, racing game, adventure game or some other type of game. 
     The GUI  902  also includes an array of frequency rank drop-down menus  924  for selection of a frequency band rank. In addition, a center frequency text editor  926  is used to set the center frequency for each frequency band. For example, the high frequency band A curve  912  has a selection menu for selecting the frequency rank  924  and the center frequency  926  of the high frequency curve A. The values in the frequency band rank drop down menus  924  may range from 0 to the number of frequency bands received from the audio analysis module  206 , with 0 being the default value of no preference for ranking, one (1) being the highest preference and the number of frequency bands received being the lowest preference. The default values of the center frequency text editor  926  are set by reading the center frequency value for each band received from the audio analysis module  206 . The center frequency can be changed for entering new center frequency values for each frequency band. 
     Radio selection box  938  allows the user to select the shaping of the impulses from a list of impulse shapers, such as but not limited to impulse shape  1 , impulse shaper  2 , and others, in order to fine tune the experience of the impulses in the immersive haptic experience. 
     In some embodiments, the array of impulses generated by the harmonic spectrogram and the percussive spectrogram can be edited separately in the GUI and then merged either by deep learning algorithms. Alternatively, the array of impulses generated by the harmonic spectrogram and the percussive spectrogram can be merged using user intervention. 
     To shape haptic characteristics, the GUI  902  allows the user multiple options such as setting a perceptual threshold value through the perceptual text box  928 , setting an impulse threshold value, which is a value that ranges from  0 . 0  to  1 . 0  by an impulse threshold text editor box  930 . 
     The haptic trigger button  932  and save button  934  allow the customizations made by the user to each of the frequency bands of the analyzed array of time frequency values and/or the analyzed array of time amplitude values through the curve editors  912 - 918 , the frequency band rank value through the frequency band rank drop down menu  924  and the center frequency value through the center frequency text editor  926  to be saved as an authored array of time frequency values, an authored array of time amplitude values and an authored array of time amplitude frequency values. In addition, customizations made by the user to an array of impulse sequence values through the impulse curve editor  920  and impulse score text box  930  are saved into an authored array of impulse sequence values. The authored the array of time amplitude values from the amplitude envelope module, and the authored array of impulse sequences from the impulse sequence of the percussive module  504 ; the authored array of time amplitude values from the amplitude envelope module and the authored array of time frequency values from the amplitude envelope associated with harmonic module  506 ; and the authored array of time amplitude frequency values associated with the residual module  508 . In addition, the actuator ID value from the actuator combo box  904 , the device ID value of the device selection combo box  906 , and perceptual threshold value of the perceptual threshold text box  928  are saved in the database  220  and provided to the transformation module  208 . Upon clicking either the save button  934 , the data mentioned above is saved as an authored audio descriptor data and other authored data, which is passed to the transformation module  210  for further processing. A trigger button  928 , captures an event, which is dispatched to the resynthesis module  214 . 
       FIG.  10    illustrates a block diagram of a transformation module in the embodiment. The transformation module  210  transforms the authored array of time frequency values and the authored array of time amplitude values from the continuous stream and the authored array of time amplitude values comprising authored array of impulse sequence and the authored array of time frequency values from impulse stream to provide an immersive haptic experience for different combinations of actuators and devices. 
     When the haptic module  110  is implemented in the electronic computing device  102 , the transformation module  210  utilizes the processor  114  and the memory  104 . However, when the haptic module  300  resides in a distributed system  150  or network  302 , the transformation module  210  has a processor  312  with the associated memory. 
     The transformation module  210  transforms the received the authored array of time frequency values and the authored array of time amplitude values from the continuous stream and the authored array of time amplitude values and the authored array of impulse sequence values to be adapted for a specific combination of the actuator  122  embedded within an electronic computing device  102 . The transformation module  210  includes a frequency transformation  1002  comprising a frequency comparison  1004 , a band equalization  1008 , and a bandwidth calculator  1010 . 
     The transformation module  210  receives the actuator specific description file and the device specific description file from the database module  220 . As discussed, the database module  220  includes the actuator specific information  222  and the device specific information  224 . The transformation module  210  queries the database module  220  utilizing the actuator ID and/or the device ID received from the authoring tool  208 . In one example, the transformation module  210  sends a request to the database module  220  for querying the actuator specific information  222  and the device specific information  224  by passing the actuator ID and/or the device ID as parameters. The database module  220  extracts the relevant information from the database module  220  and provides the results to the transformation module  210 . 
     The device specific information  224  can contain specification data or characteristic data such as the measured bandwidth, which is the acceleration over frequency response of specific actuators, such as the actuators  122  embedded within the electronic computing device  102  or the actuators  126  externally interfaced with the electronic computing device  102 . In case the device specific information  224  contains no specification data or characteristic data of a specific actuator such as the actuator  122  embedded within the electronic computing device  102  and/or the actuators  126  associated with the electronic computing device  102 , the bandwidth calculator  1010  determines the bandwidth of the electronic computing devices  102  with the embedded actuator  122 . The bandwidth calculator  1010  calculates the bandwidth using specification data provided in the actuator specific information  222 . The actuator specific information  222  includes mass of the actuator  122  or the actuator  126 , mass of the actuator  122  or the actuator  126  with additional mass attached to it, the frequency response with and without additional attached mass. The bandwidth calculator  1010  determines the frequency response of the actuator  122  or the actuator  126  along with the electronic computing device  102 . This combined bandwidth of the embedded actuator  122  within the electronic computing device is referred to as available bandwidth. To summarize, the available bandwidth is the range of frequencies at which the actuator such as the embedded actuator  122  and/or the associated actuator  126  and the electronic computing device  102  can create a vibration for an immersive haptic experience. 
     In addition, the bandwidth calculator  1010  determines the frequency and the amplitude response that can be experienced by humans. Humans can sense haptic vibrations above or below a specific threshold value within the available bandwidth of the electronic computing device  102  having embedded actuator  122 . This threshold can be predefined to a constant, for example above 0.5 g acceleration, or can be specified with the perceptual threshold  928  in the GUI  902  of the authoring tool  208 . For example, the combined bandwidth of the actuator  122  embedded in the electronic computing device  102  at which the human can feel vibration is referred to as “haptic perceptual bandwidth”. The haptic perceptual bandwidth lies in between the first threshold TH 1 , where the frequency response of the actuator  122  and the electronic computing device  102  curve just rises above a specific threshold value. This is the lower limit of the haptic perceptual bandwidth. Likewise, at another specific threshold value of the available bandwidth, the humans cannot feel the haptic experience. This threshold value is referred to as a second threshold TH 2 . 
     The haptic perceptual bandwidth, which lies between the first threshold TH 1  and the second threshold TH 2  is not fixed and can vary based on different parameters such as but not limited to individual experiences, specific thresholds to haptic vibrations at specific frequencies, specific sensitivity to haptic vibrations based on parts of the body, non-linear sensitivities to specific vibrations, and other parameters. 
     The haptic perceptual bandwidth calculated by the bandwidth calculator  1010  is provided to the frequency transformation  1002 . In some embodiments, the haptic perceptual bandwidth calculated by the bandwidth calculator  1010  is stored in the database  220 . 
     The frequency transformation  1002  includes a frequency comparison  1004 , which checks if the center frequency of each of the frequency bands received from the authored array of time frequency values and the authored array of time amplitude values from the continuous stream and the authored array of time amplitude values and the authored array of time frequency values from the impulse stream can fit within the range of the haptic perceptual bandwidth. If all of the frequencies of the frequency bands fit within the range of the haptic perceptual bandwidth, then the authored array of time frequency values and the authored array of time amplitude values from the continuous stream and the array of authored time amplitude values and the authored array of time frequency values from the impulse stream is sent directly to the band equalization  1008 . The band equalization  1008  increases or decreases the time amplitude values of the time amplitude envelope to compensate for the non-linear frequency and acceleration response curve of the actuator. For example, if a frequency that is far off from the resonant frequency has a lower acceleration force in g (gravity) then the amplitude values of the of the time amplitude envelope for this frequency is increased so that the actuator  122  creates a uniform acceleration compared to the resonant frequency when driven at that frequency. 
     However, if the frequency comparison  1008  evaluates that the authored array of time frequency values and the authored array of time amplitude values from the continuous stream and the authored array of time amplitude values and the authored array of time frequency values from the impulse stream cannot fit within the range of the haptic perceptual bandwidth, then the frequency comparison  1004  checks if the frequency band ranking has been provided. If the frequency band ranking has been provided, then the frequency comparison  1004  ranks the frequency bands in the order of the frequency band rankings. However, if no frequency band ranking is provided, the frequency transformation  1002  initiates a process of ranking the different frequency bands based on the envelope energy content. Each frequency band is ranked by determining the envelope energy content of that frequency band. The envelope energy is weighed against the distance of the energy content to the distance from the frequency of the highest acceleration in the haptic perceptual bandwidth. The frequency band with the highest acceleration value is set as the resonant frequency of the actuator  126  and the electronic computing device  102 . 
     After determination of the frequency band ranking, the frequency transformation  1002  performs a frequency mapping process, which shifts the center frequency of each of the authored array time amplitude values and the authored array of time frequency values from continuous stream by a distance, which is equal to the difference in the highest frequency acceleration value in the haptic perceptual bandwidth of the electronic computing device  102  and the frequency of the highest rank band. As such, the frequency of the highest ranked band and the frequency with the highest acceleration value in the haptic perceptual bandwidth are aligned together or superimposed on each other. In preferred embodiments, the modulus value (positive value) of the difference is taken but in other embodiments the absolute value of the difference may be taken. Subsequently, the frequency comparison  1004  checks if all the frequency bands shifted can fit within the haptic perceptual bandwidth. If each of the frequency bands, which are shifted at its centre frequency, fit within the haptic perceptual bandwidth then the frequency transformation  1002  shifts all the frequency bands at the center frequency and passes the shifted frequency bands to the band equalization  1008  for further processing. However, if all the shifted frequency bands from the continuous stream do not fit within the haptic perceptual bandwidth then the frequency transformation  1002  performs harmonic shifting of all of the bands, except the highest ranked band. The center frequency of the highest ranked band remains unaltered whereas all the other frequencies are shifted by transposing up or transposing down with a fixed constant. The fixed constant value for transposing up/transposing down is based upon the direction of the original shifting of the frequency bands. The transposing up or transposing down can in one example be performed by transposing the frequencies by one octave up or down depending upon the direction of the original shifting of the frequency bands. After the shifting of the center frequency of each of the frequency bands by one octave up or one octave down, the frequency comparison  1004  determines if the frequency bands of the continuous stream can be shifted to fit within the haptic perceptual bandwidth. If so, the bandwidth calculator  1010  passes the authored array of time amplitude values and the authored array of time frequency values from the continuous stream to the band equalization  1008  for each frequency band for further processing. Otherwise, if the frequency bands of the authored array of time amplitude values and the authored array of time frequency values from continuous stream, which are shifted by one octave do not fit within the haptic perceptual bandwidth, then the frequency bands of the authored array of time amplitude values and the authored array of time frequency values from continuous stream that do not fit within the haptic perceptual bandwidth are removed and the remaining frequency bands are passed to the band equalization  1008 . 
     The band equalization  1008  flattens and smoothes out the frequency response by boosting and/or dampening each of the frequency bands as required for haptic processing resynthesis. 
     The transformation module  208  passes an transformed continuous haptic stream and an transformed impulse haptic stream to the aggregation and file management module  212 . In addition, the transformation module  208  also provides the haptic perceptual bandwidth to the aggregation and file management module  212 . 
       FIG.  11    illustrates the different components of the aggregation and file management module in an embodiment. The aggregation and file management module  212  comprises a haptic data aggregator  1102  and a file manager  1104 . The haptic data aggregator  1102  receives the transformed continuous haptic stream and the transformed impulse haptic stream from the transformation module  210 . In addition, the aggregation and file management module  212  also receives analyzed audio data from the audio analysis module  206 . Furthermore, the aggregation and file management module  212  also receives input from the impulse processing module  204 . 
     The file manager  1104  receives the haptic data from the haptic data aggregator  1102  and converts the transformed continuous haptic stream and the transformed impulse haptic stream and the other data from database  220  into multiple computer readable file format that can be processed and synthesized by the resynthesis module  214  for producing haptic output in one or more actuators like the actuator  122  or the actuator  126 . 
     The file manager  1104  can convert the received data into different computer readable file formats for example, a text file, a JSON file, an XML, file, a CSV file, or some other file format. 
       FIG.  12    illustrates the different modules of a resynthesis module in an embodiment. The purpose of the resynthesis module  214  is to generate a haptic signal by processing the computer readable file received from the aggregation and file management module  212  to drive an actuator, such as the actuator  122  or the actuator  126 . 
     The resynthesis module  214  includes a file parser  1202 , an event receiver  1204 , an impulse playback controller  1206 , an impulse synthesizer  1210 , and one or more frequency playback controllers  1208 . 
     In some embodiments, there may be one or more frequency playback controllers  1208 , which may playback different frequencies. For example, as shown in  FIG.  12   , there are four playback controllers such as a playback controller  1208 A, a playback controller  1208 B, a playback controller  1208 C, and a playback controller  1208 D. 
     In some embodiments, there may be one or more frequency band synthesizers  1210 . As shown in  FIG.  12   , there are four frequency band synthesizers such as a frequency band synthesizer  1212 A, a frequency band synthesizer  1212 B, a frequency band synthesizer  1212 C a frequency band synthesizer  1212 D. 
     The resynthesis module  214  includes a mixer  1238 , which is configured to the actuator  122  and/or the actuator  126 . 
     Although only four playback controllers  1208  and four frequency band synthesizers  1212  are shown in the exemplary embodiment, in other embodiments, there can be fewer or more number of playback controllers  1208  and the frequency band synthesizers  1212  depending upon the number of frequency bands. 
     In embodiments, the transformed array of time amplitude values and the transformed array of time frequency values from continuous stream and the transformed array of impulses and the transformed array time amplitude values from the impulse stream are processed and transformed to match the performance characteristics of the actuator  102  embedded in the electronic computing device  102 . 
     In some embodiments, the transformed array time amplitude values and the transformed array of time frequency values from the continuous stream are used to synthesize different frequency bands and drive different playback controllers  1208 A- 1208 D. 
     The impulse playback controller  1206  is connected to an impulse synthesizer  1210 , which includes an impulse score calculator  1214 , a gain controller  1216 , an oscillator  1218  and a gain  1220 . The sequence of impulses is received at the impulse playback controller  1206 , which is passed to the impulse synthesizer  1210 . The impulse synthesizer  1210  also receives an input from the event receiver  1204 . In addition, the impulse synthesizer  1210  also receives input from the file parser  1202 . The input received by the impulse synthesizer  1210  from other modules is passed to the binary score controller  1214 . The binary score controller  1214  decides the triggering of impulse events. When the impulse event is triggered, the binary score calculator  1214  passes the amplitude values of the impulse(s) to the gain controller  1216  to control the gain  1220 . Additionally, the gain  1220  also receives the frequency values of the impulses from the oscillator  1218  and passes it to the mixer  1238  to be passed to one or more actuators  122  or the actuators  126 . 
     The playback controllers  1208 A to  1208 D receive the transformed array time frequency values and the transformed array of time amplitude values at the oscillators  1022 - 1028  for each frequency band. Simultaneously, the playback controllers  1008 A- 1008 D pass the time amplitude values to the gain  1030 - 1036 . The band synthesizer  1012 A- 1012 D is passed to a mixer  1238 , which mixes the output of different impulse synthesizer  1210  and the band synthesizers  1012 A- 1012 D to be passed to one or more actuators  122  or the actuator  126  for providing the haptic output. 
     The traditional haptic configuration comprises a single actuator that receives the haptic signal and uses a mixer to mix separate signals to output a multiplexed haptic output. In some embodiments, the continuous stream and the impulse streams may be processed to produce separate haptic output to drive at least two separate actuators. For example, one actuator  122  can be a LRA that processes the impulse sequence. The other actuator  126  may be a wideband actuator processing the continuous stream. With two different actuators separately providing haptic output for the impulse stream and the continuous stream, the haptic experience of the user. Each actuator may be tuned to different frequencies thereby providing a wide range of haptic output. It may be noted that the resonant frequency of the actuator providing a continuous stream of haptic output may be different than the haptic actuator providing the impulse sequence. 
     In some embodiments, when implementing the filter banks technique, each frequency band is linked with the one playback controller  1208 . For example, the playback controller  1208 A is linked with the frequency band  60  Hz to  100  Hz. Likewise, the playback controller  1208 D is linked with the frequency band  200  Hz to  300  Hz. 
     In embodiments, each actuator can receive its own mix of impulse and continuous signal. In some embodiments, the continuous stream of haptic output and the impulse stream of haptic output may be mixed in different proportions. In some embodiments, the mix proportion may be fixed. In some other embodiments, the mix proportion may be changed dynamically at runtime. In some embodiments, the mix proportion may be changed dynamically, which is controlled by the user interface of the authoring tool  208  or can be derived from the information stored in the database module  220  related to the actuator specific information  222  and/or the device specific information  224 . 
       FIG.  13    illustrates the process of handling an audio signal having audio silence in an embodiment. A novel way of handling the audio silence in the audio Signal is disclosed. The audio signal is illustrated in  1300 A. An audio silence is present between two audio snippets marked as  1304 . The process of handling audio silence is initiated by interpolating the audio silence between two audio snippets. The interpolated audio signal is shown by  1304 , which is a continuous audio signal after interpolation. In some embodiments, the interpolation can be performed by using the information envelope approximation and/or breakpoint analysis. At this step, the audio silence can be identified and accordingly identify the number of data points for generating the frequency envelope or the amplitude envelope. A In embodiments, the audio notes are construed as perceivable frequency differences. 
     As shown in  FIG.  1300 B , the audio notes are shown as discrete sections illustrating the frequency difference between adjacent notes. The discrete sections are combined by the user by combining the frequency and the amplitude to handle audio silence. This combination is illustrated in  FIG.  1300 C . 
     In some embodiments, the frequency line thickness is modulated by amplitude. In some embodiments, the frequency line color is modulated by amplitude. In some embodiments, the frequency discrete notes, which correspond to block height, are modulated by amplitude. In some embodiments, the frequency discrete notes block color (representing frequency difference) is modulated by amplitude. 
     In some embodiments, the frequency line thickness and frequency line color is modulated by amplitude. 
     In some embodiments, the frequency discrete notes are modulated by amplitude. In some embodiments, the frequency discrete notes may comprise block height or the block color. 
       FIG.  14    illustrates a process of converting an audio signal into a computer readable haptic file in an embodiment. The process  1400  starts at  1402  and immediately moves to  1404 . At step  1404 , the audio signal is passed to an audio preprocessor  202 . The preprocessor removes unwanted frequencies from the audio signal. The preprocessed audio signal is passed to an audio analysis module  206  at step  1408 . At step  1408 , the audio analysis module  206  performs the analysis of the preprocessed audio signal to convert it into a haptic signal. The audio analysis is performed by passing the preprocessed audio signal into a filter bank analysis or harmonic percussive source separation analysis for signal processing. In both these different processes, the spectral centroid of the signal is calculated. The output of the audio analysis is an analyzed audio signal, which is passed to the authoring tool  208  at step  1410 . The analyzed audio signal is modified/edited in the authoring tool  208 . The authoring tool  208  also receives the actuator specific information  222  and the device specific information  224  from the database module  220 . The analyzed audio signal is at least modified based on the actuator specific information  222  and the device specific information  224 . At step  1412  the authored audio signal is passed to the transformation module  210 . The transformation module  210  applies a transformation algorithm to transform the authored audio signal to fit into the haptic perceptual bandwidth of the electronic computing device  102  and the actuator  122 . The output of the transformation module  210  is a transformed audio signal. At step  1414 , the transformed audio signal is passed to the aggregation and file management module  212  for converting the transformed audio signal into a computer readable haptic file. The process  1400  terminates at step  1418 . 
       FIG.  15    illustrates a process of implementing a filter bank analysis to an audio signal in an embodiment. The process  1500  starts at  1502  and immediately moves to  1504 . At step  1504 , the preprocessed audio signal is received from the preprocessor module  202 . The preprocessed audio signal is passed to an audio analysis module  206  at step  1508 . At step  1508 , the audio analysis module  208  filters the audio signal to separate the audio signal into one or more frequency bands. For each frequency band, the spectral centroid is calculated at step  1510 . The process  1500 , at step  1512  produces an array of spectral centroid, an array of time amplitude values, and an array of time frequency values for each frequency band. In one variation of this implementation, the transients are separated and an impulse sequence is created from the array of spectral centroid, the array of time amplitude values, and the array of time frequency values. At step  1514 , the process  1500  passes the analyzed array of spectral centroid, the analyzed array of time amplitude values, and the analyzed array of time frequency values for each frequency band is passed to the authoring tool  208 . The process  1500  for analyzing the preprocessed audio signal within the audio analysis module  206  terminates at step  1518 . 
       FIG.  16    illustrates a process of implementing a harmonic percussive source separation analysis to an audio signal in an embodiment. The process  1600  starts at  1602  and immediately moves to  1604 . At step  1604 , the preprocessed audio signal is received from the preprocessor module  202 . The preprocessed audio signal is passed to an audio analysis module  206  at step  1608 . At step  1608 , the process  1600  analyzes the preprocessed audio signal using harmonic percussive source separation analysis to produce a harmonic spectrogram, a percussive spectrogram, and a residual spectrogram. In preferred embodiments, the residual spectrogram may be optional. At step  1610 , the process  1600  analyzes the harmonic spectrogram and the percussive spectrogram. At step  1612 , the process  1600  determines the spectral centroid of the harmonic spectrogram and the percussive spectrogram separately to create the array of frequency values and the array of impulse sequences for the percussive spectrogram and an array of time amplitude values and an array of time frequency values for the harmonic spectrogram. At step  1614 , the process  1600  passes the array of spectral centriod values, an array of time amplitude values, an array of time frequency values and an impulse sequence to the authoring tool  208  for authoring the haptic content. The process  1600  terminates at step  1618 . 
     The features, structures, or characteristics described throughout this specification may be combined in any suitable manner in one or more embodiments. The different embodiments and implementations shown herein and the illustrated example and for the purposes of providing what is believed to be the most useful and readily understood description of the principles and conceptual aspects in a non-limiting manner.