Source: https://patents.google.com/patent/CN101784004B/en
Timestamp: 2020-07-05 11:26:34
Document Index: 73087336

Matched Legal Cases: ['art 22', 'art 22', 'art 24', 'art 22', 'art 22', 'art 22', 'art 51']

CN101784004B - Information processing system and information processing method - Google Patents
CN101784004B
CN101784004B CN 200910259228 CN200910259228A CN101784004B CN 101784004 B CN101784004 B CN 101784004B CN 200910259228 CN200910259228 CN 200910259228 CN 200910259228 A CN200910259228 A CN 200910259228A CN 101784004 B CN101784004 B CN 101784004B
listeners head
CN 200910259228
CN101784004A (en
2008-12-16 Priority to JP2008319316A priority Critical patent/JP4849121B2/en
2008-12-16 Priority to JP2008-319316 priority
2009-12-16 Application filed by 索尼株式会社 filed Critical 索尼株式会社
2010-07-21 Publication of CN101784004A publication Critical patent/CN101784004A/en
2013-03-06 Publication of CN101784004B publication Critical patent/CN101784004B/en
238000006073 displacement Methods 0.000 claims description 36
An information processing system includes a display, a display sensor that detects a movement or a rotation of the display, a transducer unit as an earphone unit or a headphone unit, a sound processing part that processes an audio signal so as to localize a sound image in a position outside a head of a listener wearing the transducer unit and listening to sound, and an operation controller that computes an output from the display sensor to obtain a moving direction and a moving distance, or a rotation direction and a rotation angle of the display, and controls sound processing performed by the sound processing part in accordance with a result of the computation so that a positional relation between the display and the head of the listener is mapped as a positional relation between an image display surface and the head of the listener in a virtual viewing space.
The information processing system information processing method of unifying
The application comprises the relevant theme of Japanese patent application JP 2008-319316 of submitting to Japan Office with on December 16th, 2008, incorporates this paper into as a reference at this full content with this Japanese patent application.
The present invention relates to be configured to show that at display image also passes through the information processing system of earphone or headphone output sound, and the information processing method that uses this information processing system.
On watching portable display apparatus such as listening to such as sound such as music by earphone or headphone in the images such as video, this is very popular at present.
Japanese Patent Application Publication communique No.9-70094 and Japanese Patent Application Publication communique No.11-205892 have disclosed such technology: when the listener listens to music by earphone or headphone, detect the rotation of listeners head, based on the location of testing result guide sound picture (sound image), and the pre-position outside listeners head positions acoustic image.
In addition, Japanese Patent Application Publication communique No.9-93700 has disclosed a kind of when reproduced image and sound, the technology that the pre-position on display floater positions acoustic image.
Yet, for above-mentioned existing sound image localization method, because its prerequisite is display unit be the fixed installation and irremovable, therefore, when when the listener is watching such as the image on the portable display apparatus such as mobile phone, listening to sound by earphone or headphone, no matter whether show state changes, acoustic image is fixed and is positioned at the pre-position.
Particularly, though when the listener who has on earphone or headphone will such as the display unit such as mobile phone shift near, move apart or tiltedly facing to oneself the time, the sound image localized position of sound does not change yet.Therefore, for example, when utilizing portable display apparatus to listen to sound, can not provide the sort of sense of reality that experiences when seeing a film on sit in front at the cinema seat, seat, back or the seat, screen side.
Therefore expectation is controlled Sound image localization, thereby when the listener listens to music by earphone or headphone and watches image on the portable display apparatus in his or she hand, mobile and when rotating this display unit simultaneously, the sense of reality when moving on to another seat from a seat when seeing a film in just can experiencing seemingly at the cinema.
The information processing system of the embodiment of the invention comprises: display, and it moves or rotates with respect to listeners head; Show transducer, it is configured to detect described movement or the described rotation of described display; Converter unit, it is configured to headphone unit or headset unit; Acoustic processing section, it is configured to audio signal is processed, in order in the position that has on described converter unit and listen to outside the listeners head of sound acoustic image is positioned; And arithmetic and control unit, it is configured to calculating from the output of described demonstration transducer, with moving direction and the displacement that obtains described display, perhaps rotation direction and rotational angle, and the acoustic processing of being undertaken by described acoustic processing section according to result of calculation control is in order to make position relationship between described display and the listeners head be mapped as picture display face in the virtual audio-visual space and the position relationship between the listeners head.
Information processing system is the information processing system according to above-described embodiment in accordance with another embodiment of the present invention, described information processing system also comprises the conversion transducer, and described conversion transducer is attached on the described converter unit and is configured to detect movement or the rotation of listeners head.Described arithmetic and control unit is configured to calculating from the output of described demonstration transducer and the output of described conversion transducer, with moving direction and the displacement that obtains described display, perhaps rotation direction and rotational angle, and the moving direction of described listeners head and displacement, perhaps rotation direction and rotational angle, and the acoustic processing of being undertaken by described acoustic processing section according to result of calculation control is in order to make position relationship between described display and the listeners head be mapped as described picture display face in the virtual audio-visual space and the position relationship between the listeners head.
Information processing system with embodiment of the invention of said structure positions acoustic image, thereby when the listener shifts near display, move apart or tiltedly face toward oneself, in virtual audio-visual space, the listener is near picture display face, away from picture display face or shift to the left side of the picture display face that tilts with respect to picture display face or the position on right side.
Sense of reality when therefore, Sound image localization provides when the listener sees a film at the cinema seemingly and to have moved on to another seat from a seat.
Because loud speaker was as main loudspeaker before most music source adopted, thereby shifts near display and can increase volume, remove display and can reduce volume, therefore, do not need control members such as key and switch, information processing system can also play the effect of volume adjusting interface.
As mentioned above, according to the embodiment of the invention, when the listener is listening to sound by earphone or headphone, and move and when rotating display unit the sense of reality when Sound image localization can provide when the listener sees a film at the cinema seemingly and to move on to another seat from a seat in the image on the portable display apparatus in watching his or she hand.
Fig. 1 is the schematic diagram of example of the external structure of embodiment of the invention information processing system;
Fig. 2 is the block diagram of the syndeton of embodiment of the invention information process unit;
Fig. 3 shows the schematic diagram of the example of virtual audio-visual space;
Fig. 4 is the block diagram for the example of the structure of Sound image localization;
Fig. 5 shows the schematic diagram of the example of initial condition;
The schematic diagram of the example when Fig. 6 shows according to the embodiment mobile display;
Fig. 7 shows the position of listener in virtual audio-visual space shown in Figure 6 and the schematic diagram of direction;
Fig. 8 shows the schematic diagram that rotates the example of display according to embodiment;
Fig. 9 shows the position of listener in virtual audio-visual space shown in Figure 8 and the schematic diagram of direction;
Figure 10 shows the schematic diagram that moves and rotate the example of display according to embodiment;
Figure 11 shows the position of listener in virtual audio-visual space shown in Figure 10 and the schematic diagram of direction;
Figure 12 is the flow chart of the example of a series of processing of being undertaken by the arithmetic and control unit in the information process unit of embodiment;
Figure 13 shows for the explanation of calculating displacement and rotational angle according to embodiment;
Figure 14 shows the schematic diagram of example of the headphone unit of another embodiment of the present invention;
Figure 15 is the block diagram of syndeton of the information process unit of another embodiment;
Figure 16 shows the schematic diagram that moves and rotate the example of display and listeners head according to another embodiment;
Figure 17 shows the position of listener in virtual audio-visual space shown in Figure 16 and the schematic diagram of direction;
Figure 18 is the flow chart of the example of a series of processing of carrying out of the arithmetic and control unit by the information process unit of another embodiment;
Figure 19 shows for the explanation of calculating displacement and rotational angle according to another embodiment; And
Figure 20 is the schematic diagram of the information processing system of the embodiment of the invention.
1, embodiment: Fig. 1～Figure 13
Embodiments of the invention show the listener and do not move or rotate, and only are the situations that display moves and/or rotates.
1-1, system configuration: Fig. 1～Fig. 4
The external structure of 1-1-1, system: Fig. 1
Fig. 1 shows the example of external structure of the information processing system of the present embodiment.
Information processing system 100 shown in Figure 1 comprises information process unit 10 and headphone unit 50.
Information process unit 10 can reproduce such as images such as videos with such as sound such as music, and comprise in appearance displays 11 such as liquid crystal display or organic electroluminescent (Electroluminescent, EL) display and have in addition operation keys and the operating portion 12 of operation driver plate.
Headphone unit 50 comprises left earphone section 60 and right earphone section 70, and is connected with being connected respectively to be connected with right earphone section with left earphone section 60 from the branch line 56 of the end bifurcated of line 55.
Although not shown among Fig. 1, line 55 other ends are with plug, and described plug inserts in the socket that is arranged in the information process unit 10, thereby makes headphone unit 50 and information process unit 10 wired connections.
The syndeton of 1-1-2, system: Fig. 2
Fig. 2 shows the syndeton of information process unit 10.
Information process unit 10 comprises bus 14, not only operating portion 12 is connected to this bus 14, and central processing unit (central processing unit, CPU) 15, read-only memory (read onlymemory, ROM) 16, random access memory (random access memory, RAM) 17 and nonvolatile memory 19 also are connected to this bus 14.
In read-only memory (ROM) 16, write in advance and will pass through various computer programs and the necessary fixed data that central processing unit (CPU) 15 is carried out.Random access memory (RAM) 17 plays the working region of central processing unit (CPU) 15.
Central processing unit (CPU) 15, read-only memory (ROM) 16 and random access memory (RAM) 17 form arithmetic and control unit 21, this arithmetic and control unit 21 carries out with the movement of display 11 and rotates relevant calculating, and according to hereinafter described result of calculation control Sound image localization.
Nonvolatile memory 19 is assembled in the information process unit 10, perhaps be attached on the information process unit 10, and storage is such as the view data such as video with such as voice datas such as music.
Image processing part 22 is connected with the sound handling part and is connected with bus 14, and each image processing part 22 and sound handling part 24 comprise central processing unit (CPU) 15, read-only memory (ROM) 16 and random access memory (RAM) 17.
Image processing part 22 will convert analog picture signal to such as view data such as videos from what nonvolatile memory 19 read.If view data is compressed, then image processing part 22 is at first to its decompression.
24 pairs in acoustic processing section carries out hereinafter described Sound image localization from what nonvolatile memory 19 read such as voice datas such as music.If voice data is compressed, then acoustic processing section 24 is at first to its decompression.
Picture signal from image processing part 22 converts display drive signals to and is supplied to display 11 by drive circuit section 23.
Convert simulated audio signal from the digital audio data about acoustic processing section 24 to by digital analog converter (digital to analog converter, DAC) 25 and 26
Audio signal about after the conversion is amplified by audio amplifier circuit 27 and 28, and is supplied to the transducer 61 and 71 of headphone unit 50 left and right sides.
Transducer 61 and 71 will convert sound to such as audio signals such as music.
In this example, information process unit 10 also is provided with acceleration transducer 31 and gyrosensor 32, acceleration transducer 31 is for detection of the movement of display 11, it is the movement of information process unit 10, and gyrosensor 32 is for detection of the rotation of display 11, i.e. the rotation of information process unit 10.
Particularly, acceleration transducer 31 detects the acceleration of the movement on the direction of two orthogonal axles (X-axis and Y-axis) on the described datum level hereinafter, and gyrosensor 32 detects the angular speed that rotates around the axle vertical with described datum level (Z axis).
Output signal from acceleration transducer 31 and gyrosensor 32 is sampled by analog-digital converter (analog to digital converter, ADC) 33 and 34 respectively, is converted into numerical data, and sends bus 14 to.
1-1-3, virtual audio-visual space: Fig. 3
, suppose to exist such as virtual audio-visual spaces such as virtual movie institutes also by the information process unit 10 of headphone unit 50 output sounds for show image at display 11.Fig. 3 shows the example of virtual audio-visual space.
Virtual audio-visual space 1 in this example is the coffin (plane that is parallel to the paper among Fig. 3) on the datum level, wherein, be furnished with picture display face 2, center loudspeaker 3, left speaker 4 and right loud speaker 5 in the place ahead of listener, and be furnished with respectively loud speaker 6 and 7 on left side and the right side in close the place ahead.
The quantity of loud speaker and layout thereof only represent example, and any amount of loud speaker can be arranged in any position.
Picture display face 2 be have image show thereon, as the panel of projection screen or display.
Position Po is the center of virtual audio-visual space 1, and the state of the listeners head 9 that represents with solid line show listeners head 9 at position Po place the state in the face of picture display face 2.
The listener moves to position Pf is equivalent to move on to the front in real cinema seat from position Po, and moves to position Pb is equivalent to move on to the back in real cinema seat from position Po.
The listener moves to position P1 is equivalent to move on to the left side in real cinema seat from position Po, and moves to position Pr is equivalent to move on to the right side in real cinema seat from position Po.
The in a lateral direction extension of X-axis in virtual audio-visual space 1, the longitudinal direction of Y-axis in virtual audio-visual space 1 extends, and Z axis is extending perpendicular to the direction of datum level (plane that is parallel to the paper among Fig. 3).
1-1-4, Sound image localization: Fig. 4
Fig. 4 shows when the virtual audio-visual space 1 of hypothesis as shown in Figure 3 the time, the example of the structure of the Sound image localization that is undertaken by the acoustic processing section 24 in the information process unit 10.
Audio signal SC, SL, SR, SE and SF are the digital audio datas of each sound channels of the virtual speaker 3,4,5 from be arranged in virtual audio-visual space 1 shown in Figure 3,6 and 7 outputs respectively.If data are compressed, then export the digital audio data that decompresses.
Audio signal SC is supplied to digital filter 43L and 43R, and audio signal SL is supplied to digital filter 44L and 44R, and audio signal SR is supplied to digital filter 45L and 45R.
Audio signal SE is supplied to digital filter 46L and 46R, and audio signal SF is supplied to digital filter 47L and 47R.
Digital filter 43L is to carrying out convolution by will be from the position of loud speaker 3 converting the impulse response that time domain produces to the transfer function H CL of listeners head 9 left ears.
Digital filter 43R is to carrying out convolution by will be from the position of loud speaker 3 converting the impulse response that time domain produces to the transfer function H CR of listeners head 9 auris dextras.
Digital filter 44L is to carrying out convolution by will be from the position of loud speaker 4 converting the impulse response that time domain produces to the transfer function H LL of listeners head 9 left ears.
Digital filter 44R is to carrying out convolution by will be from the position of loud speaker 4 converting the impulse response that time domain produces to the transfer function H LR of listeners head 9 auris dextras.
Digital filter 45L is to carrying out convolution by will be from the position of loud speaker 5 converting the impulse response that time domain produces to the transfer function H RL of listeners head 9 left ears.
Digital filter 45R is to carrying out convolution by will be from the position of loud speaker 5 converting the impulse response that time domain produces to the transfer function H RR of listeners head 9 auris dextras.
Digital filter 46L is to carrying out convolution by will be from the position of loud speaker 6 converting the impulse response that time domain produces to the transfer function H EL of listeners head 9 left ears.
Digital filter 46R is to carrying out convolution by will be from the position of loud speaker 6 converting the impulse response that time domain produces to the transfer function H ER of listeners head 9 auris dextras.
Digital filter 47L is to carrying out convolution by will be from the position of loud speaker 7 converting the impulse response that time domain produces to the transfer function H FL of listeners head 9 left ears.
Digital filter 47R is to carrying out convolution by will be from the position of loud speaker 7 converting the impulse response that time domain produces to the transfer function H FR of listeners head 9 auris dextras.
Pass through add circuit 41 additions from the audio signal of digital filter 43L, 44L, 45L, 46L and 47L output.Pass through add circuit 42 additions from the audio signal of digital filter 43R, 44R, 45R, 46R and 47R output.
Convert simulated audio signal from the audio signal of add circuit 41 outputs to by DAC shown in Figure 2 25.Audio signal through conversion is enlarged into left audio signal by audio amplifier circuit 27, then is supplied to transducer 61.
Convert simulated audio signal from the audio signal of add circuit 42 outputs to by DAC shown in Figure 2 26.Audio signal through conversion is enlarged into right audio signal by audio amplifier circuit 28, then is supplied to transducer 71.
1-2, information processing method: Fig. 5～Figure 13
According to described embodiment, Sound image localization is controlled, thereby when display 11 moves or rotates, make mobile or rotate after display 11 and the position relationship between the listeners head 9 be mapped as picture display face 2 in the virtual audio-visual space 1 and the position relationship between the listeners head 9.
1-2-1, initial condition: Fig. 5
In order in this way Sound image localization to be controlled, be necessary to set initial condition.
Fig. 5 shows the example of the initial condition of setting in actual audio-visual space.
When the listener uses information processing system 100 to watch image and listens to music, listener's operating operation section 12, information process unit 10 is set as initial condition, and in this initial condition, display 11 is positioned at the position that is certain distance and direction with the listener.
Fig. 5 shows such situation: the listener utilizes the information process unit 10 in his or she hand to set initial conditions in the face of display 11, therefore, display 11 be positioned at frontal on the Do place, position of position Ho Lo separated by a distance of listeners head 9.
In this case, for information process unit 10, extend and the face that intersects with the panel of display 11 at a predetermined angle is datum level from the panel of display 11 in a lateral direction, the in a lateral direction extension of the panel of X-axis on datum level, Y-axis is extended in the direction vertical with X-axis, and Z axis extends in the direction vertical with datum level.
Acceleration transducer 31 shown in Figure 2 detects the acceleration of the movement on X-direction and the Y direction, and gyrosensor 32 detects the angular speed of the rotation on the Z-direction.
Although the initial distance Lo between display 11 and the listeners head 9 chooses wantonly, when the people watches display floater in his or she hand, the general approximately 30cm of distance.
Initial condition refers to such state: the pre-position of for example center Po of listener in virtual audio-visual space 1 as shown in Figure 3, and watch and listen to such as image and sound such as films.
Therefore, when the position relationship between display 11 and the listeners head 9 is in predefined initial condition, Sound image localization is controlled, thereby made the listener can hear sound from virtual speaker 3～7 in position Po shown in Figure 3 and direction.
1-2-2, when display moves: Fig. 6 and Fig. 7
In the first method of the present embodiment, the listener makes display 11 mobile in X-direction or Y direction.
Fig. 6 shows such situation: 11m is represented such as Reference numeral, the listener with display 11 from above-mentioned initial condition along X-axis positive direction displacement Dx, and along Y-axis negative direction displacement Dy.
The X-axis positive direction is the right direction of panel, and the X-axis negative direction is the left direction of panel, and the Y-axis positive direction is to leave the direction of listeners head 9, and the Y-axis negative direction is the direction near listeners head 9.
Position Do is the initial position of display 11, and position Dm is the position after display 11 moves.
Distance L m is as the display 11m of the display 11 after mobile and the distance between the listeners head 9.If initial distance Lo is set as for example 30cm, then distance L m can calculate with formula shown in Figure 6 (1).
Arithmetic and control unit 21 in the information process unit 10 is by to carrying out integration twice from the X-direction of acceleration transducer 31 output and each acceleration on the Y direction, and calculation display 11 is at the displacement Dx on the X-axis and the displacement Dy on Y-axis.
In addition, arithmetic and control unit 21 in the information process unit 10 is selected and the processing parameter of definite Sound image localization, thereby makes display 11m after mobile and the position relationship between the listeners head 9 be mapped as picture display face 2 in the virtual audio-visual space 1 and the position relationship between the listeners head 9.
A kind of method for Mapping and Converting comprises calculates Qx=KDx, Qy=KDy, and wherein, K is the conversion coefficient on the X-direction, also is the conversion coefficient on the Y direction, and Qx is the displacement on X-axis, and Qy is the displacement on Y-axis.
Because compare with distance L o in the actual audio-visual space with the scope that listener's hand can reach in actual audio-visual space farthest, distance between the scope of virtual audio-visual space 1 and picture display face 2 and the center Po is enough large, so conversion coefficient K should be greater than 1.
Display 11 in actual audio-visual space along X-axis positive direction displacement Dx with along the fact of Y-axis negative direction displacement Dy, be equivalent to listeners head 9 in virtual audio-visual space 1 along X-axis negative direction displacement Qx with along the fact of Y-axis positive direction displacement Qy.
Therefore, as shown in Figure 7, calculate from center Po along X-axis negative direction displacement Qx and along the position of Y-axis positive direction displacement Qy, with the position Pm of this position as listeners head 9 in the virtual audio-visual space 1.
Picture display face 2 from virtual audio-visual space 1, position Pm are positioned at respect to the Y-axis negative direction along on the direction of the angle [alpha] of formula (2) expression that clockwise rotates Fig. 6.
Another kind method comprises the position Pm that adopts distance L m and angle [alpha] to calculate the listeners head 9 in the virtual audio-visual space 1.
That is to say, in this case, for from picture display face 2 read fortune for the Y-axis negative direction along clockwise rotate on the direction of angle [alpha], leave the point apart from lm from the center of picture display face 2 horizontal directions, this position Pm as the listeners head 9 in the virtual audio-visual space 1 is calculated, and described is the product of distance L m and conversion coefficient K apart from lm.
Can determine conversion coefficient K by the degree of depth Cy of (longitudinal direction) on the width C x or Y direction that consider (horizontal direction) on the X-direction in the virtual audio-visual space 1.
For example, the brachium of supposing the people is 50cm, and the distance L m maximum in actual audio-visual space between display 11 and the listeners head 9 is 50cm.
The maximum of supposing distance L m is Lmmax, when considering degree of depth Cy:
lm:Lm＝Cy:Lmmax????????(5)
lm＝Cy×Lm/Lmmax???????(6)
In addition, when considering width C x:
lm:Lm＝Cx/2:Lmmax??????(7)
lm＝Cx×Lm/2×Lmmax????(8)
When 1-2-3, rotation display: Fig. 8 and Fig. 9
When the listener makes display 11 when Z axis rotates, adopt the second method of the present embodiment.
Fig. 8 shows such situation: 11r is represented such as Reference numeral, the listener with display 11 from initial condition shown in Figure 5, take position Do as center of rotation around Z axis along (nearside of paper plane) seen from the top counter clockwise direction rotational angle φ.
Arithmetic and control unit 21 in the information process unit 10 is by to calculating rotational angle φ from angular speed integrations gyrosensor 32 output, that rotate around Z axis.
In addition, arithmetic and control unit 21 in the information process unit 10 is selected and definite Sound image localization processing parameters, thereby makes display 11r after the rotation and the position relationship between the listeners head 9 be mapped as picture display face 2 in the virtual audio-visual space 1 and the position relationship between the listeners head 9.
Particularly, display 11 in actual audio-visual space in the counterclockwise direction the fact of rotational angle φ corresponding to listeners head 9 fact of rotational angle φ along clockwise direction in virtual audio-visual space 1.
Therefore, in this case, as shown in Figure 9, for from picture display face 2 read fortune for the Y-axis negative direction along clockwise rotate on the direction of angle φ, leave the point apart from lo from the center of picture display face 2 horizontal directions, this position Pm as the listeners head 9 in the virtual audio-visual space 1 is calculated, and described is the product of distance L o and conversion coefficient K apart from lo.
Listeners head 9 towards on the direction in the face of the center of the horizontal direction of picture display face 2.
1-2-4, movement and when rotating display: Figure 10 and Figure 11
When the listener makes display 11 move and rotate, adopt the third method of the present embodiment.
Figure 10 shows an example, in this example, shown in Reference numeral 11mr, the listener make display 11 from initial condition shown in Figure 5 displacement Dx on the X-axis positive direction and on the Y-axis negative direction displacement Dy, and make at display 11 in the counterclockwise direction around Z axis rotational angle φ.
In other words, in this case, display 11 is mobile and as shown in Figure 8 rotation as shown in Figure 6.
In this case, as shown in figure 11, for from picture display face 2 read fortune for the Y-axis negative direction along clockwise rotate angle beta=(on the direction of φ+α), leave from the center of picture display face 2 horizontal directions that (point of=K * Lm) calculates this position Pm as the listeners head 9 in the virtual audio-visual space 1 apart from lm.
The processing of 1-2-5, s operation control: Figure 12 and Figure 13
Figure 12 shows the example of a series of processing of being undertaken by the arithmetic and control unit 21 in the information process unit 10 of the present embodiment.
In this example, in step 111, according to above-mentioned listener's operating and setting initial condition.
Then, in step 112, to sampling from the output signal of the diaxon of acceleration transducer 31 with from the output signal of gyrosensor 32, and convert numerical data to, thereby obtain representing display 11 in the data of the acceleration of X-direction and Y direction movement and represent that display 11 is around the data of the angular speed of Z axis rotation.
In step 113, utilize formula shown in Figure 13 (11), (12) and (13) calculation display 11 the displacement Dx on the X-direction, on Y direction displacement Dy and around the rotational angle φ of Z axis.
In step 114, according to result of calculation, determine the filter coefficient of digital filter 43L, 43R, 44L, 44R, 45L, 45R, 46L, 46R, 47L and 47R shown in Figure 4.
In step 115, according to the filter coefficient of determining, acoustic processing section 24 carries out Sound image localization.
In step 116, judge whether above-mentioned a series of processing should stop, and except when above-mentioned a series of processing for example by outside the termination of listener's terminating operation, is processed and returned step 112 from step 116, repeating step 112 is to the processing of step 115.
2, another embodiment: Figure 14～Figure 19
Another embodiment of the present invention shows such situation: not only make display mobile and/or rotation as among the above-mentioned embodiment, and also mobile and/or rotation of listener.
2-1, system configuration: Figure 14 and Figure 15
According to another embodiment, for example shown in Figure 1, information processing system 100 comprises information process unit 10 and headphone unit 50.
This another embodiment part similar to the above embodiments is also that information process unit 10 is seen from the outside and comprises display 11 and operating portion 12.
In addition, according to another embodiment, headphone unit 50 disposes the movement that can detect listeners head 9 or the transducer of rotation.Figure 14 shows example.
Transducer 61 and grid 63 and the line cover on the other end (cord bushing) 64 that left earphone section 60 is attached with on inside casing 62 1 ends.
Acceleration transducer 65, gyrosensor 66 and housing 67 are attached on the part of left earphone section 60 of ear outside.Receiver 69 is attached on the part of left earphone section 60 of ear inside.
The same with left earphone section 60, right earphone section 70 is attached with transducer 71 and grid 73 and the cover of the line on the other end 74 on inside casing 72 1 ends.
Housing 77 is attached on the part of right earphone section 70 of ear outside.Receiver 79 is attached on the part of right earphone section 70 of ear inside.
Acceleration transducer 65 detects mobile acceleration on the direction of orthogonal two axles (X-axis and Y-axis) on the described datum level hereinafter, and gyrosensor 66 detects the angular speed that rotates around the axle vertical with described datum level (Z axis).
As shown in figure 15, in information process unit 10, except the structure of embodiment shown in Figure 2, ADC 35 and the ADC 36 that will convert from the output signal of the acceleration transducer 65 of headphone unit 50 and gyrosensor 66 respectively numerical data to also are connected to bus 14.
According to another embodiment, for example, supposing has virtual audio-visual space 1 shown in Figure 3, and the acoustic processing section in the information process unit 10 24 carries out Sound image localization shown in Figure 4.
2-2, information processing method: Figure 16～Figure 19
According to another embodiment, information process unit 10 is according to listener's operating and setting initial condition.For example, described initial condition is state shown in Figure 5.
According to another embodiment, display 11 and listener's movement and/or the combination of rotation have following situation:
(a) listener's mobile display 11 and mobile his or her head;
(b) listener's mobile display 11 and rotate his or her head;
(c) listener rotates display 11 and mobile his or her head;
(d) listener rotates display 11 and rotates his or her head;
(e) listener moves and rotates display 11 and moves and rotate his or her head.
Under any circumstance, all Sound image localization is controlled, thereby made display 11 in the actual audio-visual space and the position relationship between the listeners head 9 be mapped as picture display face 2 in the virtual audio-visual space 1 and the position relationship between the listeners head 9.
Figure 16 shows situation (e), and wherein, the listener moves and rotates display 11 and moves and rotate his or her head.
Particularly, in this case, display 11 is mobile and rotation as shown in Figure 10, and listeners head 9 is displacement Hx on the X-axis positive direction, displacement Hy on the Y-axis negative direction, and around Z axis rotational angle θ along clockwise direction, the direction of rotation of the rotation direction of listeners head 9 and display 11.
Position Do, distance L o, position Dm, distance D x, distance D y and rotational angle φ and Fig. 5, Fig. 6, Fig. 8 and corresponding position, distance and rotational angle shown in Figure 10 are identical.
In this case, position Ho is the initial position of listeners head 9, and position Hm is the position after listeners head 9 moves.
As described in above-mentioned embodiment, display 11 the displacement Dx on the X-axis and at the displacement Dy on the Y-axis by to calculating for twice from the X-direction of acceleration transducer 31 output and each integrated acceleration on the Y direction.
Listeners head 9 the displacement Hx on the X-axis and at the displacement Hy on the Y-axis by to calculating for twice from the X-direction of acceleration transducer 65 output and each integrated acceleration on the Y direction.
As described in above-mentioned embodiment, the rotational angle φ of display 11 is by calculating the angular speed integration from gyrosensor 32 outputs.
The rotational angle θ of listeners head 9 is by calculating the angular speed integration from gyrosensor 66 outputs.
If initial distance Lo is set to for example 30cm, after then display 11 and listeners head 9 moved and rotate, the distance L m between display 11mr and the listeners head 9 can calculate with formula shown in Figure 16 (3).Angle [alpha] shown in Figure 16 represents with formula shown in Figure 16 (4).
Arithmetic and control unit 21 in the information process unit 10 is selected and the processing parameter of definite Sound image localization, thus make aforesaid movement and rotate after display 11mr and the position relationship between the listeners head 9 be mapped as picture display face 2 in the virtual audio-visual space 1 and the position relationship between the listeners head 9.
Listeners head 9 in actual audio-visual space along clockwise direction the fact of rotational angle θ corresponding to listeners head 9 also fact of rotational angle θ along clockwise direction in virtual audio-visual space 1.
Therefore, in this case, as shown in figure 17, for from picture display face 2 read fortune for the Y-axis negative direction along clockwise rotate angle (on the direction of φ+θ), leave from the center of picture display face 2 horizontal directions that (point of=K * Lm) calculates this position Pm as the listeners head 9 in the virtual audio-visual space 1 apart from lm.
Figure 18 shows the example of a series of processing of being undertaken by the arithmetic and control unit 21 in the information process unit 10 of another embodiment.
In this example, in step 121, as mentioned above, according to listener's operating and setting initial condition.
Then, in step 122, to the output signal from the diaxon of acceleration transducer 31, output signal from gyrosensor 32, sample from the output signal of the diaxon of acceleration transducer 65 with from the output signal of gyrosensor 66, and convert numerical data to, thereby obtain representing that display 11 is in the data of the acceleration of X-direction and Y direction movement, expression display 11 is around the data of the angular speed of Z axis rotation, represent listeners head 9 in the data of the acceleration of X-direction and Y direction movement and represent that listeners head 9 is around the data of the angular speed of Z axis rotation.
In step 123, utilize formula shown in Figure 19 (11), (12) and (13) calculation display 11 at the displacement Dx on the X-direction, the displacement Dy on Y direction and the rotational angle φ that rotates around Z axis, and utilize formula shown in Figure 19 (21), (22) and (23) to calculate listeners head 9 at the displacement Hx on the X-direction, the displacement Hy on Y direction and the rotational angle θ that rotates around Z axis.
In step 124, according to result of calculation, determine the filter coefficient of digital filter 43L, 43R, 44L, 44R, 45L, 45R, 46L, 46R, 47L and 47R shown in Figure 4.
In step 125, according to the filter coefficient of determining, acoustic processing section 24 carries out Sound image localization.
In step 126, judge whether above-mentioned a series of processing should stop, and except when above-mentioned a series of processing for example by outside the termination of listener's terminating operation, is processed and returned step 122 from step 126, repeating step 122 is to the processing of step 125.
3, other embodiment: Figure 20
As shown in figure 20, information processing system 100 can comprise display unit 80, information process unit 90 and headphone unit 50.In this case, expectation is by for example bluetooth (registered trade mark ) etc. radio communication display unit 80 is connected with information process unit 90, and information process unit 90 is connected with headphone unit 50.
Information process unit 90 is stored in view data and music data in the hard disk etc. as home server, and carries out the acoustic processing that image is processed and comprised above-mentioned Sound image localization.
Display unit 80 comprises display 11, operating portion 12, for detection of the acceleration transducer of the movement of display 11 with for detection of the gyrosensor of the rotation of display 11 etc., and the output signal of autobiography sensor sends information process unit 90 in the future.
Headphone unit 50 comprises the circuit part 51 that is provided with battery, radio communication module and volume control, and in order as among above-mentioned another embodiment, to process movement and/or the rotation of listeners head 9, be provided with acceleration transducer and gyrosensor in left earphone section 60 or right earphone section 70.
Even when information processing system 100 comprises information process unit 10 and headphone unit 50 as shown in Figure 1, information process unit 10 also can be connected with headphone unit 50 by radio communication.
Converter unit is not limited to headphone unit, also can be the headphone unit.
It will be appreciated by those skilled in the art that according to designing requirement and other factors, can in the scope of the appended claim of the present invention or its equivalent, carry out various modifications, combination, inferior combination and change.
1. information processing system, described information processing system comprises:
Display, it moves or rotates with respect to listeners head;
Show transducer, it is configured to detect described movement or the described rotation of described display;
Converter unit, it is configured to headphone unit or headset unit;
Acoustic processing section, it is configured to audio signal is processed, in order in the position that has on described converter unit and listen to outside the listeners head of sound acoustic image is positioned; And
Arithmetic and control unit, it is configured to calculating from the output of described demonstration transducer, with moving direction and the displacement that obtains described display, perhaps rotation direction and rotational angle, and the acoustic processing of being undertaken by described acoustic processing section according to result of calculation control is in order to make position relationship between described display and the listeners head be mapped as picture display face in the virtual audio-visual space and the position relationship between the listeners head.
2. information processing system according to claim 1, described information processing system also comprises the conversion transducer, described conversion transducer is attached on the described converter unit and is configured to detect movement or the rotation of listeners head;
Wherein, described arithmetic and control unit is configured to calculating from the output of described demonstration transducer and the output of described conversion transducer, with moving direction and the displacement that obtains described display, perhaps rotation direction and rotational angle, and the moving direction of described listeners head and displacement, perhaps rotation direction and rotational angle, and the acoustic processing of being undertaken by described acoustic processing section according to result of calculation control is in order to make position relationship between described display and the listeners head be mapped as described picture display face in the virtual audio-visual space and the position relationship between the listeners head.
3. information processing system according to claim 1, wherein, described information processing system comprises information process unit and described converter unit, and described information process unit has described display, described demonstration transducer, described acoustic processing section and described arithmetic and control unit.
4. information processing system according to claim 1, wherein, described information processing system comprises the display unit with described display and described demonstration transducer, the information process unit with described acoustic processing section and described arithmetic and control unit and described converter unit.
5. information processing method that is undertaken by information processing system, described information processing system comprises: display, it moves or rotates with respect to listeners head; Show transducer, it is configured to detect described movement or the described rotation of described display; Converter unit, it is configured to headphone unit or headset unit; And acoustic processing section, it is configured to audio signal is processed, in order in the position that has on described converter unit and listen to outside the listeners head of sound acoustic image is positioned,
Described information processing method may further comprise the steps:
To calculating from the output of described demonstration transducer, with moving direction and the displacement that obtains described display, perhaps rotation direction and rotational angle; And
The acoustic processing of being undertaken by described acoustic processing section according to result of calculation control is in order to make position relationship between described display and the listeners head be mapped as picture display face in the virtual audio-visual space and the position relationship between the listeners head.
6. information processing method according to claim 5, described information processing system also comprises the conversion transducer, described conversion transducer is attached on the described converter unit and is configured to detect movement or the rotation of described listeners head, described information processing method is further comprising the steps of: to calculating from the output of described conversion transducer, with moving direction and the displacement that obtains described listeners head, perhaps rotation direction and rotational angle;
Wherein, described control step comprises: according to from the result of calculation of the output of described demonstration transducer and the acoustic processing to being undertaken by described acoustic processing section from the result of calculation control of the output of described conversion transducer, in order to make position relationship between described display and the listeners head be mapped as described picture display face in the virtual audio-visual space and the position relationship between the listeners head.
CN 200910259228 2008-12-16 2009-12-16 Information processing system and information processing method CN101784004B (en)
JP2008319316A JP4849121B2 (en) 2008-12-16 2008-12-16 Information processing system and information processing method
JP2008-319316 2008-12-16
CN101784004A CN101784004A (en) 2010-07-21
CN101784004B true CN101784004B (en) 2013-03-06
ID=42112209
CN 200910259228 CN101784004B (en) 2008-12-16 2009-12-16 Information processing system and information processing method
US (1) US8644531B2 (en)
EP (1) EP2200349B1 (en)
JP (1) JP4849121B2 (en)
CN (1) CN101784004B (en)
AT (1) AT515899T (en)
WO2013105413A1 (en) * 2012-01-11 2013-07-18 ソニー株式会社 Sound field control device, sound field control method, program, sound field control system, and server
CN103052018B (en) * 2012-12-19 2014-10-22 武汉大学 Audio-visual distance information recovery method
CN103037301B (en) * 2012-12-19 2014-11-05 武汉大学 Convenient adjustment method for restoring range information of acoustic images
JP2014143470A (en) * 2013-01-22 2014-08-07 Sony Corp Information processing unit, information processing method, and program
CN106154231A (en) * 2016-08-03 2016-11-23 厦门傅里叶电子有限公司 The method of sound field location in virtual reality
US10277973B2 (en) * 2017-03-31 2019-04-30 Apple Inc. Wireless ear bud system with pose detection
CN1158047A (en) * 1995-09-28 1997-08-27 索尼公司 image/audio reproducing system
CN101065990A (en) * 2004-09-16 2007-10-31 松下电器产业株式会社 Sound image localizer
JP3687099B2 (en) * 1994-02-14 2005-08-24 ソニー株式会社 Video signal and audio signal playback device
JPH10230899A (en) * 1997-02-24 1998-09-02 Motoya Takeyama Man-machine interface of aerospace aircraft
JP3994296B2 (en) 1998-01-19 2007-10-17 ソニー株式会社 Audio playback device
JP3435141B2 (en) * 2001-01-09 2003-08-11 松下電器産業株式会社 Sound image localization device, conference device using sound image localization device, mobile phone, audio reproduction device, audio recording device, information terminal device, game machine, communication and broadcasting system
JP2006294032A (en) * 2002-09-05 2006-10-26 Sony Computer Entertainment Inc Display system, display control device, display apparatus, display method, and user interface device
JP2006165845A (en) * 2004-12-06 2006-06-22 Alpine Electronics Inc Video-audio apparatus
JP2006186904A (en) * 2004-12-28 2006-07-13 Mitsumi Electric Co Ltd Head set
WO2006107074A1 (en) * 2005-04-05 2006-10-12 Matsushita Electric Industrial Co., Ltd. Portable terminal
JP2008219759A (en) * 2007-03-07 2008-09-18 Navitime Japan Co Ltd Portable media content reproduction system, portable media content reproduction apparatus and media content distribution server
2008-12-16 JP JP2008319316A patent/JP4849121B2/en active Active
2009-12-10 AT AT09252762T patent/AT515899T/en not_active IP Right Cessation
2009-12-10 US US12/634,999 patent/US8644531B2/en active Active
2009-12-10 EP EP20090252762 patent/EP2200349B1/en active Active
2009-12-16 CN CN 200910259228 patent/CN101784004B/en active IP Right Grant
JP2010147529A (en) 2010-07-01
CN101784004A (en) 2010-07-21
JP4849121B2 (en) 2012-01-11
AT515899T (en) 2011-07-15
EP2200349A1 (en) 2010-06-23
US20100150355A1 (en) 2010-06-17
EP2200349B1 (en) 2011-07-06
US8644531B2 (en) 2014-02-04
US10129681B2 (en) 2018-11-13 Calibrating listening devices
JP5642851B2 (en) 2014-12-17 Hearing aid
CN106412772B (en) 2020-01-07 Camera driven audio spatialization
EP2389016B1 (en) 2013-07-10 Individualization of sound signals
US9363602B2 (en) 2016-06-07 Method and apparatus for providing virtualized audio files via headphones
JP3385725B2 (en) 2003-03-10 Audio playback device with video
US7817806B2 (en) 2010-10-19 Sound pickup method and apparatus, sound pickup and reproduction method, and sound reproduction apparatus
EP0695109B1 (en) 2011-07-27 Device for reproducing video signal and audio signal
RU2591179C2 (en) 2016-07-10 Method and system for generating transfer function of head by linear mixing of head transfer functions
US9544706B1 (en) 2017-01-10 Customized head-related transfer functions