Abstract:
The present invention discloses methods, apparatus and systems for individualizing music, audio and speech adaptively, intelligently and interactively according to a listener&#39;s personal hearing ability, unique hearing preference, characteristic feedback, and real-time surrounding environment.

Description:
CROSS-REFERENCE TO RELATED APPLICATIONS 
     NOT APPLICABLE 
     STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT 
     NOT APPLICABLE 
     FIELD OF THE INVENTION 
     The present invention relates generally to the fields of intelligent audio, music and speech processing. It also relates to individualized equalization curves, individualized delivery of music, audio and speech, and interactively customized music content. More particularly, the present invention relates to methods, apparatus and systems for individualizing audio, music and speech adaptively, intelligently and interactively according to a listener&#39;s personal hearing ability, unique hearing preference, characteristic feedback, and real-time surrounding environment. 
     BACKGROUND OF THE INVENTION 
     For home theaters, personal listening systems, recording studios, and other sound systems, signal processing plays a critical role. Among many signal processing techniques, equalization is commonly used to alter the amount of energy allocated in different frequency bands to make a sound more sensational, or to render said sound with new properties. When a sound engineer sets up a sound system, the system as a whole is commonly equalized in frequency domain to compensate for equipment distortion, room acoustics, and most importantly a listener&#39;s preference. Therefore, equalization is a listener-dependent task, and the best equalization relies on adaptive and intelligent individualization. Similarly, spatial audio and speech enhancement, among others, require adaptive and intelligent individualization to achieve best perceptual quality and satisfy personal hearing ability. 
     Currently, rapid growth of computational ability of personal listening systems increases signal processing power significantly, which makes it feasible to individualize personal sound systems by low system-level computational complexity. 
     SUMMARY OF THE INVENTION 
     Disclosed herein are methods, apparatus and systems for individualizing audio, music and speech adaptively, intelligently and interactively. One aspect of the present invention involves finding a set of parameters of a personal listening system that best fits a listener, wherein an automated test is conducted to determine the best set of parameters. During the test, the present invention characterizes personal hearing preference, hearing ability, and surrounding environment to search, optimize and adjust said personal listening system. Another aspect of the invention provides an adaptive and intelligent search algorithm to automatically assess a listener&#39;s hearing preference and hearing ability in a specific listening environment with reliable convergence. The advantages of the present invention include portability, repeatability, independency of music and speech content, and straightforward extensibility into existing personal listening systems. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG. 1  is an explanatory block diagram showing an individualized personal listening system of the present invention. 
         FIG. 2  is an explanatory block diagram showing a signal processing framework of an embodiment of the present invention. 
         FIG. 3  is an explanatory block diagram showing a detection component of hearing preference of the present invention. 
         FIG. 4  is an explanatory block diagram showing an individualized personal listening system for sound externalization of the present invention. 
     
    
    
     DETAILED DESCRIPTION OF THE INVENTION 
     As used herein, the term “plurality” shall mean two or more than two. The term “another” is defined as a second or more. The terms “including” and “having” are open ended. The term “or” is interpreted as inclusive or meaning any one or any combination. 
     Reference throughout this document to “one embodiment”, “certain embodiments”, and “an embodiment” or similar terms means that a particular element, function, step, act, feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present invention. Thus, the appearances of such phrases in various places are not necessarily all referring to the same embodiment. Furthermore, the disclosed elements, functions, steps, acts, features, structures, or characteristics can be combined in any suitable manner on one or more embodiments without limitation. An exception will occur only when a combination of said elements, functions, steps, acts, features, structures, or characteristics, are in some way inherently mutually exclusive. 
     In one embodiment, referring to  FIG. 1 , an incoming sound input is adjusted by an automatic fluctuation control unit (AFCU)  1010  before entering a windowing unit (WDU)  1020  and a zero padding unit  1180 . When the output of said zero padding unit  1180  is transformed into a plurality of time-frequency bins by a forward transform unit  1160 , said time-frequency bins pass a cepstrum unit  1170  to output a cepstrum. Said cepstrum is processed by at least one cepstrum-domain lifter  1150  to output a cepstrum vector into an adaptive classification unit (ACU)  1090 . Additionally, the output of said forward transform unit  1160  is directed to a weighted fusion unit  1140  that merges adjacent time-frequency bins according to non-linear psychoacoustic-based auditory tuning curves. Accordingly, the output of said weighted fusion unit  1140  provides auditory system based representation of said incoming sound. Additionally, the output of said weighted fusion unit is employed by a long-term high-order moment calculation unit (LHMCU)  1030  to compute variance, skewness and kurtosis in a long-term manner. Furthermore, the output of said weighted fusion unit is also employed by a short-term high-order moment calculation unit (SHMCU)  1060  to calculate short-term variance, skewness and kurtosis. Said long-term and short-term variances, skewnesses and kurtosises are directed to the ACU  1090 . The output of said weighted fusion unit passes a multi-block weighted averaging unit (MBWAU)  1120  to suppress a plurality of undesired components. Said MBWAU delivers a first output and a second output, wherein said first output is a long-term mean value  1100  and said second output is a short-term mean value  1110 . Said long-term and short-term mean values are delivered to said ACU  1090 . Said ACU  1090  utilizes said cepstrum vector, said long-term and short-term mean values, said long-term and short-term variances, said long-term and short-term skewnesses, and said long-term and short-term kurtosises, to classify current instantaneous signal into a beat category or a non-beat category. Said classification leads to a beat signal  1080 . In parallel, said ACU  1090  adaptively updates said AFCU  1010 , said WDU  1020 , and a plurality of weighting coefficients  1130 . Said weighting coefficients  1130  control the MBWAU  1120  to compute said long-term and short-term mean values. Said beat signal  1080  controls an individualized auditory enhancer (IAE)  1050  to enhance auditory perception in accordance to a listener&#39;s human input unit  1040 . At the same time, said beat signal  1080  drives at least one individualized multimodal enhancer (IME)  1070 . The IME  1070  activates at least one tactile actuator, vibrator, visual displayer, or motion controller, wherein said tactile actuator, said vibrator, said visual displayer, or said motion controller, stimulates human sensory modalities. 
     In broad embodiment, the present invention comprises filtering an original audio signal by manipulating a magnitude response and a phase response, assigning said phase response to compensate for a group delay according to a result of a hearing test, searching for the best set of audio parameters, and individualizing said audio adaptively and intelligently for an individual. 
     In another embodiment, an assessment process is added to confirm reliability of the best EQ curve chosen by testing a listener, and an evaluation result is automatically obtained in regard to reliability. 
     In one embodiment, said best EQ curves can be transferred to another generic equalizer so that a listener can listen to an equalized song through said generic equalizer. 
     In one embodiment, said best EQ curves are encoded to programmable earphones, headphones, headsets or loudspeakers so that said earphones, said headphones, said headsets or said loudspeaker becomes individualized and suitable for a plurality of music songs. 
     In another embodiment, a vocal separation module serves as a front end, separates audio material into a plurality of streams including a vocal stream, a plurality of instrumental streams and a background environmental stream, applies an individualized set of parameters that are obtained through a hearing test to each stream, and mixes said equalized streams together. 
     In one embodiment, referring to  FIG. 2 , an incoming sound input is sent to an input adapting unit  2170  for adapting to quality and amplitude of said sound input. A first output of said input adapting unit  2170  is directed to a direct current removing unit  2160  to remove direct current components. A second output of said direct current removing unit  2160  is delivered to a multiplexing unit  2150  to pre-process multi-dimensional properties of said sound input for a forward transform. A windowing unit  2140  is applied to conduct a window function to a third output of said multiplexing unit  2150 . Zeros are padded to a fourth output of said windowing unit  2140  through a first zero padding unit  2120 . A forward transform is performed on a fourth output of said first zero padding unit  2120  by a first forward transform unit  2110 , whereas said first forward transform unit  2110  generates a first stream. Said first stream is delivered to a beat sensing unit  2180 , wherein said beat sensing unit  2180  extracts a beat signal from said first stream. Said beat signal is sent to a visual animation unit  2190 , wherein said visual animation unit  2190  stimulates individual visual perception. An individual motion sensing unit  2220  is employed to detect an individual motion, wherein said individual motion unit  2220  stimulates an individual motion conversion unit  2210 . A converted motion waveform is conveyed from said individual motion conversion unit  2210  to said visual animation unit  2190 , a spatial data loading unit  2200 , an equalization curve searching unit  2240 , and a filter shaping unit  2230 , wherein said spatial data loading unit  2200  loads a transformed frequency response of a spatial impulse response into a channel arranging unit  2070 , said equalization curve searching unit  2240  searches for an equalization curve for an individual, and said filter shaping unit  2230  adjusts a response contour of a function combining unit  2030 . A fifth output of a test result converter unit  2020  is sent to said function combining unit  2030 , wherein said test result converter unit  2020  extracts a sixth output of a hearing test unit  2010 . A combined stream is provided from said test result converter  2020 , said equalization curve searching unit  2240 , and said filter shaping unit  2030  to a first reverse transform unit  2040 , wherein said first reverse transform unit  2040  conducts a reverse transform. A seventh output of said first reverse transform unit  2040  is delivered to a second zero padding unit  2050 , wherein said zero padding unit adds zeros to said seventh output of said reverse transform unit  2040 . A second stream is combined from said spatial data loading unit  2200 , said beat sensing unit  2180 , and a second forward transform unit  2060 , wherein said forward transform unit  2060  conducts a forward transform on an eighth output of said zero padding unit  2050 . Said second stream is delivered to a magnitude and phase manipulating unit  2080 , wherein a channel separating unit  2100  converts said first stream to a plurality of channels, and said magnitude and phase manipulating unit  2080  adjusts magnitude and phase of said channels. Finally, a ninth output of said magnitude and phase manipulating unit  2080  is sent to a second reverse transform unit  2090  for auditory perception enhancement. 
     In another embodiment, referring now to  FIG. 3 , an incoming sound input from an environment monitoring unit  3010  is extracted, wherein said environment monitoring unit  3010  stimulates an environment analyzing unit  3020  to generate a first stream, a second stream, a third stream, a fourth stream, a fifth stream, a sixth stream and a seventh stream. Sequential order of a plurality of stimulation sounds is arranged in a sound sequencing unit  3160 , wherein said first stream controls said sound sequencing unit  3160 . A first sound is generated in a sound generating unit  3030 , wherein said second stream determines a plurality of characteristics of said first sound. Bandwidth of said stimulation sounds is adjusted in a bandwidth adjusting unit  3140 , wherein a group delay unit  3130  sound receives a first output of said bandwidth adjusting unit  3140 , applies phase spectrum that matches a group delay to generate a first signal, and sends said first signal to a sound mixing unit  3120 . Said first signal is mixed with said first sound to generate a mixed signal according to said third stream. A binaural signal is provided for a binaural strategy unit  3110  based on said mixed signal, wherein said fourth stream determines a plurality of characteristics of said binaural signal for a sound manipulating unit  3060 . An ear interface unit  3100  is driven according to a first output of a human interface unit  3090 , wherein said sound manipulating unit  3060  delivers a third sound to said human interface unit  3090 . Said fifth stream is processed in a user-data analyzing unit  3070 , wherein said user-data analyzing unit  3070  combines a second output of said human interface unit  3090  with said fifth stream to generate a confidence level. Said confidence level is sent to a confidence level unit  3200  for storage. Said sixth stream is delivered to a result output unit  3080 , wherein said result output unit  3080  converts said sixth stream for visual stimulation. An indication is provided to an individual listener through said seventh stream on a plurality of characteristics of time-frequency analysis. A plurality of functions of a platform is identified through a platform identifying unit  3190 , wherein said platform identifying unit  3190  transmits said functions to a sound calibrating unit  3180 . Finally, said sound mixing unit  3120  is adjusted according to a calibration mode unit  3170 , wherein said calibration mode unit  3170  is changed by said human interface unit  3090 . 
     In broad embodiment, referring now to  FIG. 4 , a multi-dimensional reality audio is individualized and delivered, wherein the overall processing is decomposed into a plurality of joint processing units. First, a sensory analysis unit  4100  directs an incoming sound to extract a first stream and classify said sound into one category out of a plurality of categories. Said first stream is processed by a sound combining unit  4010 , wherein said sound combining unit  4010  maps a dimension of said first stream to another dimension of a second stream. Said second stream is provided to a sound externalization unit  4020 , wherein said sound externalization unit  4020  filters said second stream to increase externalization auditory effect. The output of said sound externalization unit  4020  is transformed through a forward transform unit  4030 . Furthermore, a first output of said forward transform unit  4030  is processed by a sound spatialization unit  4110  for a spatial effect according to said category that is determined by said sensory analysis unit  4100 . Additionally, a first control signal is obtained through a human input unit  4090  from a listener, wherein said human input unit  4090  converts said first control signal to a second control signal for said sound externalization unit  4020  through a personalization structuring unit  4080 . A second output of said sound spatialization unit  4110  passes a reverse transform unit  4040 . A magnitude and phase manipulating unit  4060  provides a third control signal to adjust magnitude responses and phase responses of said first output of said forward transform unit  4030  through a personalization structuring unit  4080 . A fourth control signal from said personalization structuring unit  4080  is delivered to a dynamic database unit  4070  to extract an individual interaural spatialization response, wherein said individual interaural spatialization response is processed to improve a spatial resolution by a multiple-dimensional interpolation unit  4050 . 
     Multi-modal perception throughout the present invention enhances individual auditory experience. The present invention derives stimuli for various modalities, wherein the derivation targets the fundamental attributes of said stimuli: modality, intensity, location and duration, and aims at affecting multi-cortical areas. 
     While the invention has been described in connection with various embodiments, it should be understood that the invention is capable of further modifications. This application is intended to cover any variations, uses or adaptation of the invention following, in general, the principles of the invention, and including such departures from the present disclosure as come within the known and customary practice within the art to which the invention pertains. 
     While the foregoing written description of the invention enables one of ordinary skill to make and use what is considered presently to be the best mode thereof, those of ordinary skill will understand and appreciate the existence of variations, combinations, and equivalents of the specific embodiment, method, and examples herein. The invention should therefore not be limited by the above described embodiments, methods, and examples, but by all embodiments and methods within the scope and spirit of the invention as described herein.