Patent Description:
An apparatus is proposed that receives tactile data transmitted from a transmission device and that causes vibrators provided on a jacket or the like to vibrate in a vibration pattern based on the tactile data to thereby allow the user wearing the jacket to enjoy a sense of touch with a sense of reality (see PTL <NUM>). PTL <NUM> describes a granular synthesis that can be used to create a dynamic haptic effect. PTL <NUM> describes a vest with tactile interface devices configured to provide information transfer through somatosensation.

Incidentally, installing more vibrators on the jacket to provide a higher sense of reality is considered in the technique of PTL <NUM>.

However, if the number of vibrators is increased, the number of channels for the tactile data transmitted to the jacket may increase, and the transmission time of the tactile data may increase according to the number of increased channels.

The present disclosure has been made in view of the circumstances, and particularly, the present disclosure enables to efficiently transmit and receive tactile data of multiple channels.

The above problem is solved by information processing apparatuses in accordance with the appended claims.

According to one aspect of the present disclosure, tactile data of multiple channels can efficiently be transmitted and received.

Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the attached drawings. Note that in the present specification and the drawings, the same signs are provided to constituent elements with substantially the same functional configurations, and the description will not be repeated.

Hereinafter, the embodiments of the present technique will be described. The embodiments will be described in the following order.

A summary of the present disclosure will be described with reference to <FIG>.

<FIG> illustrates a summary of a configuration of a content reproduction system that reproduces content including tactile data according to the present disclosure.

A content reproduction system <NUM> of <FIG> includes a transmission apparatus <NUM>, a reception apparatus <NUM>, a display apparatus <NUM>, and a vest-type tactile presentation apparatus <NUM> worn by a user <NUM>. A wired or wireless communication path not illustrated is reserved for each component, and the components can transfer data to each other.

The transmission apparatus <NUM> supplies content including video data, audio data, and tactile data to the reception apparatus <NUM>.

The reception apparatus <NUM> supplies content including video data and audio data to the display apparatus <NUM> and causes the display apparatus <NUM> to display images and output sound to allow the user to view the content. Further, at the same time, the reception apparatus <NUM> supplies the tactile data to the tactile presentation apparatus <NUM> worn by the user <NUM> to realize the tactile presentation corresponding to the tactile data.

The vest-type tactile presentation apparatus <NUM> includes a plurality of vibration units 13a at positions in close contact with various parts of the body of the user <NUM>. The vest-type tactile presentation apparatus <NUM> causes the vibration units 13a to independently vibrate based on the tactile data of each of the vibration units 13a supplied from the reception apparatus <NUM> to thereby make a tactile presentation (provide a stimulus) at each corresponding part of the body of the user <NUM>.

The video data, the audio data, and the tactile data included in the same content correspond to each other. Therefore, the user <NUM> wearing the tactile presentation apparatus <NUM> can enjoy the video and the sound of the content displayed by the display apparatus <NUM> and enjoy watching the content with a higher sense of reality through the tactile presentation by the plurality of vibration units 13a based on the tactile data corresponding to the flow of the content. For example, the user can enjoy the tactile presentation in synchronization with the elapsed time of the reproduction of the displayed video and audio content.

Incidentally, although ten vibration units 13a are depicted in the tactile presentation apparatus <NUM> in <FIG>, more vibration units 13a can be provided in reality.

The higher the number of vibration units 13a is, the higher the sense of reality can be. However, the number of channels for the tactile data supplied to the tactile presentation apparatus <NUM> increases with an increase in the number of vibration units 13a. This increases the transmission time for transmitting the tactile data of multiple channels corresponding to the plurality of vibration units 13a from the transmission apparatus <NUM> to the reception apparatus <NUM>.

As a result, with the increase in the transmission time of the tactile data, the timing of the actual vibration and the tactile presentation by the vibration units 13a of the tactile presentation apparatus <NUM> may be delayed from the image or the sound.

Therefore, to increase the vibration units 13a to allow expressing the tactile data with a higher sense of reality for the user <NUM> wearing the tactile presentation apparatus <NUM>, the time required for transmitting and receiving the tactile data needs to be reduced to suppress the delay to allow the plurality of vibration units 13a to make the tactile presentation at appropriate timing in synchronization with the video and the audio content.

Therefore, in the present disclosure, an estimator obtained based on frequency analysis, pattern matching, or machine learning is used to make an estimate to divide the tactile signal corresponding to the tactile data into a base tactile signal group, and the division result is used to convert the tactile data into encoded data to transmit and receive the encoded data. As a result, the tactile data can be expressed by simple parameters defining the divided base tactile signals, and the amount of data can be compressed. The delay in the tactile presentation can also be suppressed in the transmission and the reception of the tactile data of multiple channels, and as a result, the tactile data can be reproduced in synchronization with the video and the audio content.

Note that in the present specification, the tactile signal is a waveform signal expressing vibration generated by the vibration of the vibration unit 13a of the tactile presentation apparatus <NUM>, and the tactile data is data for causing the vibration unit 13a to generate vibration corresponding to the waveform signal expressed by the tactile signal. In other words, the tactile data can also be considered as a command or a parameter supplied to the vibration unit 13a of the tactile presentation apparatus <NUM> to realize the vibration of the waveform signal corresponding to the tactile signal. Therefore, the vibration unit 13a vibrates based on the tactile data, and the waveform signal input to the vibration unit 13a and the waveform signal generated by the vibration are the tactile signals. Similarly, the display apparatus <NUM> displays images and outputs sound based on the image data and the audio data.

In addition, the base tactile signal group is a group of a plurality of tactile signals as bases that can be combined to generate one tactile signal. For example, in a case where two optional base tactile signals among the plurality of base tactile signals belonging to the base tactile signal group are extracted, the two extracted base tactile signals are in an orthogonal relationship with each other.

In other words, the base tactile signals belonging to the base tactile signal group are independent from each other and are not similar. That is, the base tactile signals are tactile signals not overlapping with each other.

In addition, for example, approximately several hundred types of base tactile signals are prepared as the base tactile signals belonging to the base tactile signal group, and there may be base tactile signals with the same length. It is desirable that the base tactile signal group is a group of tactile signals including specific waveform signals with various lengths, and the tactile signals are independent from each other.

A tactile signal TS corresponding to predetermined tactile data can be expressed as a sum of a plurality of base tactile signals, and for example, as illustrated in <FIG>, the tactile signal TS can be divided into divided information DI including a base tactile signal BTS1, a base tactile signal BTS2, a base tactile signal BTS3,. belonging to the base tactile signal group.

Note that although the divided information DI of <FIG> includes one base tactile signal BTS1 of one type, one base tactile signal BTS2 of another type, one base tactile signal BTS3 of still another type,. , the divided information DI may include a plurality of same base tactile signals.

In addition, although the base tactile signal is a tactile signal including a specific waveform signal at predetermined time, the divided information DI obtained by dividing the tactile signal TS may be divided in a state in which a gain as an increase or a decrease in the amplitude, an offset in the time direction, and the like are applied to each base tactile signal.

In the present disclosure, parameters including an index for identifying each base tactile signal and a gain, an offset in the time direction, and the like to be applied to each base tactile signal are used for each of the base tactile signal BTS1, the base tactile signal BTS2, the base tactile signal BTS3,. included in the divided information D1 to encode the tactile signal.

More specifically, each of the base tactile signal BTS1, the base tactile signal BTS2, the base tactile signal BTS3,. included in the divided information D1 is expressed by parameters, such as a base tactile signal index, a gain, an offset (time code), and a length (of time), and the signals are combined to generate combined information MI. In the present disclosure, the combined information MI is the encoded data of the tactile signal TS.

Here, the base tactile signal index is an identifier of the target base tactile signal. The gain is a gain added in the entire period of the target base tactile signal. The offset (time code) is an offset time from predetermined timing of the start of the tactile presentation by the vibration unit 13a based on the target base tactile signal, and the length is a length (period of time) of the target base tactile signal.

In such a way, in the present disclosure, the tactile signal TS is divided based on the base tactile signal group, and the parameters of each of the divided base tactile signals are used to encode the signal. As a result, the tactile signal TS corresponding to any tactile data is compressed into the encoded data including the combined information MI based on the base tactile signal group.

Decoding of the encoded data into the tactile data is realized by executing a process opposite the encoding method.

That is, the encoded data is the combined information MI in which the parameters, such as a base tactile signal index, a gain, an offset, and a length, of each of the base tactile signal BTS1, the base tactile signal BTS2, the base tactile signal BTS3,. included in the divided information D1 are combined. Therefore, the encoded data as the combined information MI is divided to extract the parameters defining each of the base tactile signal BTS1, the base tactile signal BTS2, the base tactile signal BTS3,. stored in advance, and a process based on the extracted parameters is applied to the base tactile signal BTS1, the base tactile signal BTS2, the base tactile signal BTS3,. to restore the divided information DI.

Further, the waveform signals of the base tactile signal BTS1, the base tactile signal BTS2, the base tactile signal BTS3,. restored as the divided information DI by using the parameters are added to restore an original tactile signal TS' (tactile signal TS' close to the tactile signal TS), and the tactile data is generated based on the restored tactile signal TS'.

That is, the tactile signal TS is converted into the divided information DI based on the base tactile signal group, and the parameters defining each of the base tactile signals included in the divided information DI are used to encode the divided information DI into the combined information MI. The divided information DI is restored from the combined information MI as the encoded data, and the tactile signal TS' is restored. Therefore, the tactile data can highly efficiently be compressed, transmitted, and received.

As a result, even when the tactile data of multiple channels is transmitted and received, the series of processes can be realized at a high speed. Even when the plurality of vibration units 13a is vibrated, the delay in the timing of vibration can be suppressed. Therefore, the vibration units 13a can be vibrated in synchronization with the video and the audio content.

Note that the types of the base tactile signals belonging to the base tactile signal group may be increased or decreased, or the lengths of the base tactile signals may be adjusted according to the quality of the tactile presentation based on the tactile data to be reproduced or the throughput of the tactile presentation apparatus <NUM>.

That is, in a case where the throughput of the tactile presentation apparatus <NUM> is high so that the quality of the tactile presentation based on the tactile data can be increased, the types of the base tactile signals included in the base tactile signal group may be increased, or the lengths of the base tactile signals may be shortened. In such a way, the tactile data can be divided into more types of base tactile signals, and even high-quality multi-channel tactile data can be transmitted and received at a high speed although the processing load is high. This can suppress the delay in the tactile presentation.

Conversely, in a case where the throughput of the tactile presentation apparatus <NUM> is low so that highspeed processing needs to be prioritized even by reducing the quality of the tactile presentation based on the tactile data, the number of base tactile signals included in the base tactile signal group may be reduced, or the lengths of the base tactile signals may be extended. In such a way, the kinds of base tactile signals to be divided can be reduced, and the processing load can be reduced. Multi-channel tactile data can be transmitted and received at a high speed even if the throughput of the tactile presentation apparatus <NUM> is low, and the delay in the tactile presentation can be suppressed.

Note that the tactile signal TS before encoding and the tactile signal TS' restored after encoding may be different signals in a strict sense. This is because when the input tactile signal TS is divided into the divided information DI, the tactile signal TS is divided based on a limited base tactile signal group. Even when the divided information DI is used to restore the tactile signal, a complete tactile signal TS may not be obtained. In this case, the tactile signal TS' close to the tactile signal TS is reproduced.

Therefore, in the following, in a case where the tactile signals before encoding and after encoding need to be particularly distinguished, the input tactile signal before encoding will also be referred to as an input tactile signal, and the tactile signal to be output restored after encoding will also be referred to as an output tactile signal. In addition, similarly, the tactile data corresponding to the input tactile signal will also be referred to as input tactile data, and the tactile data corresponding to the output tactile signal will also be referred to as output tactile data as necessary.

Next, a configuration example of a content reproduction system that reproduces content including image data, audio data, and tactile data according to the present disclosure will be described with reference to <FIG>.

A content reproduction system <NUM> of <FIG> includes a transmission apparatus <NUM>, a reception apparatus <NUM>, a tactile presentation apparatus <NUM>, and a display apparatus <NUM>.

The transmission apparatus <NUM> has a configuration corresponding to the transmission apparatus <NUM> in <FIG> and transmits the data of the content including the tactile data in addition to the image data and the audio data to the reception apparatus <NUM> through a network represented by a LAN (Local Area Network) based on wireless communication, wired communication, or the like.

The reception apparatus <NUM> has a configuration corresponding to the reception apparatus <NUM> in <FIG> and transmits the tactile data included in the content data to the tactile presentation apparatus <NUM> through a network represented by a LAN (Local Area Network) based on near field communication such as Bluetooth (registered trademark), wireless communication such as WiFi, wired communication, or the like. The reception apparatus <NUM> causes tactile presentation units 53a-<NUM> to 53a-n including vibration units or the like to vibrate according to the tactile data to make a tactile presentation.

In addition, the reception apparatus <NUM> outputs the image data and the audio data included in the content data to the display apparatus <NUM> through wireless communication, such as WiFi, or wired communication to cause the display apparatus <NUM> to display the data and output the sound.

The tactile presentation apparatus <NUM> has, for example, a configuration corresponding to the tactile presentation apparatus <NUM> in <FIG> and includes a plurality of tactile presentation units 53a-<NUM> to 53a-n corresponding to the vibration units 13a. The tactile presentation apparatus <NUM> is worn by the user, and the tactile presentation units 53a-<NUM> to 53a-n come into contact with predetermined parts of the body of the user. Note that in the following, the tactile presentation units 53a-<NUM> to 53a-n will be simply referred to as tactile presentation units 53a in a case where the tactile presentation units 53a-<NUM> to 53a-n do not have to be particularly distinguished, and other components will also be similarly called.

The tactile presentation apparatus <NUM> causes the plurality of tactile presentation units 53a to vibrate to execute a tactile presentation at a predetermined part of the body of the user to provide a predetermined stimulus based on the tactile data supplied from the reception apparatus <NUM> and corresponding to the reproduction state of the content. As a result, when the user views the content reproduced by the display apparatus <NUM>, the tactile presentation units 53a-<NUM> to 53a-n make a tactile presentation at a predetermined part of the body according to the reproduced scene, and the user can enjoy viewing the content with a sense of reality.

Note that although the tactile presentation units 53a are vibration units that generate vibration in time series to provide a stimulus to a predetermined part of the user in the example described here, the operation may be another operation based on the tactile data, and for example, electrical muscle stimulation or pressure may be applied in time series.

The display apparatus <NUM> corresponds to the display apparatus <NUM> of <FIG> and is a display including an LCD (Liquid Crystal Display) or an organic EL (Electro Luminescence:). In addition, the display apparatus <NUM> includes a speaker. The display apparatus <NUM> displays images based on the image data supplied from the reception apparatus <NUM> and outputs sound based on the audio data.

The tactile presentation apparatus <NUM> may be, for example, a vest-type tactile presentation apparatus 53A as illustrated in <FIG>.

The vest-type tactile presentation apparatus 53A of <FIG> is provided with tactile presentation units 53a-<NUM> to 53a-<NUM> that vibrate to make a tactile presentation at a part on the front side of the body of the user.

The tactile presentation units 53a-<NUM> to 53a-<NUM> include, for example, piezoelectric element actuators or the like and vibrate based on the tactile signal supplied from the reception apparatus <NUM> to make a tactile presentation to the user based on vibration stimulation.

In addition, the tactile presentation apparatus <NUM> can also be, for example, a wristband-type tactile presentation apparatus 53B as illustrated in <FIG>.

The tactile presentation apparatus 53B of <FIG> is installed at a part near the wrist of the user.

The tactile presentation apparatus 53B is provided with tactile presentation units 53a-<NUM> to 53a-<NUM> and is a wristband-type device with a function of vibrating to make a tactile presentation.

The tactile presentation apparatus 53B is provided with the tactile presentation units 53a-<NUM> to 53a-<NUM> that vibrate to make a tactile presentation at a part near the wrist of the user.

The tactile presentation units <NUM>-<NUM> to <NUM>-<NUM> include, for example, piezoelectric element actuators or the like and vibrate based on the supplied tactile signal to make a tactile presentation to the user based on vibration stimulation.

Further, the tactile presentation apparatus <NUM> can also be, for example, a portable-device-type tactile presentation apparatus 53C, such as a game device, as illustrated in <FIG>. In this case, the display apparatus <NUM> for the tactile presentation apparatus <NUM> may be integrated.

That is, as illustrated in <FIG>, the tactile presentation apparatus 53C is a portable device held by the hands of the user to operate. For example, the user holds both ends of the tactile presentation apparatus 53C by hand to play a game or the like by operating buttons B1 to B4 provided on the tactile presentation apparatus 53C while viewing the images of the content displayed on a display unit D, such as a liquid crystal panel, that functions similarly to the display apparatus <NUM> and listening to the sound output from a speaker, a headphone, or the like not illustrated.

Tactile presentation units 53a-<NUM> and 53a-<NUM> that make a tactile presentation to the user according to the images of the content displayed on the display unit D and the output sound are provided at end parts inside of the tactile presentation apparatus 53C.

The tactile presentation units 53a-<NUM> and 53a-<NUM> include, for example, piezoelectric element actuators or the like and vibrate based on the tactile data supplied from the reception apparatus <NUM> to make a tactile presentation to the user based on vibration stimulation. Note that in the examples illustrated in <FIG>, the tactile presentation apparatus <NUM> may be a tactile presentation apparatus <NUM> that makes a tactile presentation of a category different from the vibration stimulation.

Next, a configuration example of the transmission apparatus <NUM> will be described with reference to <FIG>.

The transmission apparatus <NUM> includes a control unit <NUM>, a communication unit <NUM>, a storage unit <NUM>, an input unit <NUM>, an output unit <NUM>, a drive <NUM>, and a removable storage medium <NUM> that are connected to each other through a bus <NUM> so that data and programs can be transmitted and received.

The control unit <NUM> includes a processor and a memory and controls the entire operation of the transmission apparatus <NUM>. The control unit <NUM> includes a content processing unit <NUM> and a tactile data processing unit <NUM>.

The content processing unit <NUM> reads content data including image data, audio data, and tactile data stored in the storage unit <NUM> and controls the communication unit <NUM> to transmit the content data to the reception apparatus <NUM>. In this case, the content processing unit <NUM> supplies the tactile data to the tactile data processing unit <NUM> to cause the tactile data processing unit <NUM> to convert the tactile data into encoded data and then causes the tactile data processing unit <NUM> to transmit the converted encoded data to the reception apparatus <NUM> along with the image data and the audio data.

The tactile data processing unit <NUM> includes an encoding unit <NUM>. The tactile data processing unit <NUM> causes the encoding unit <NUM> to use the tactile data in the content data supplied from the content processing unit <NUM> as input tactile data to convert the input tactile data into encoded data and outputs the encoded data to the content processing unit <NUM>. Note that the encoding method of the tactile data executed by the encoding unit <NUM> will be described in detail later with reference to <FIG> and <FIG>.

The communication unit <NUM> is controlled by the control unit <NUM> and uses wired (or wireless (not illustrated)) communication to transmit and receive various data and programs to and from the reception apparatus <NUM> through a communication network represented by a LAN (Local Area Network) or the like.

The storage unit <NUM> is controlled by the control unit <NUM> and includes an HDD (Hard Disk Drive), an SSD (Solid State Drive), a semiconductor memory, or the like. The storage unit <NUM> writes or reads various data including the content data and programs.

The input unit <NUM> includes an input device, such as a keyboard and a mouse, for the user to input an operation command and supplies various input signals to the control unit <NUM>.

The output unit <NUM> is controlled by the control unit <NUM> and outputs and displays a supplied processing operation screen or an image of a processing result on a display device not illustrated.

The drive <NUM> reads and writes data to and from the removable storage medium <NUM>, such as a magnetic disk (including a flexible disk), an optical disk (including a CD-ROM (Compact Disc-Read Only Memory) and a DVD (Digital Versatile Disc)), a magneto-optical disk (including an MD (Mini Disc)), and a semiconductor memory.

Next, a configuration example of the reception apparatus <NUM> will be described with reference to <FIG>.

The reception apparatus <NUM> includes a control unit <NUM>, a communication unit <NUM>, a storage unit <NUM>, an input unit <NUM>, an output unit <NUM>, a drive <NUM>, and a removable storage medium <NUM> that are connected to each other through a bus <NUM> so that data and programs can be transmitted and received.

The control unit <NUM> includes a processor and a memory and controls the entire operation of the reception apparatus <NUM>. The control unit <NUM> includes a content processing unit <NUM> and a tactile data processing unit <NUM>.

The content processing unit <NUM> acquires the content data including the encoded data of encoded image data, audio data, and input tactile data supplied from the transmission apparatus <NUM> through the communication unit <NUM>. Further, the content processing unit <NUM> controls the communication unit <NUM> to output the image data and the audio data to the display apparatus <NUM> to cause the display apparatus <NUM> to display images and output sound. In addition, the content processing unit <NUM> supplies the encoded data to the tactile data processing unit <NUM> and causes the tactile data processing unit <NUM> to restore the data as output tactile data. The content processing unit <NUM> then supplies the output tactile data to the tactile presentation apparatus <NUM> and causes the tactile presentation units 53a to make a tactile presentation based on the output tactile data.

The tactile data processing unit <NUM> includes a decoding unit <NUM>. The tactile data processing unit <NUM> causes the decoding unit <NUM> to decode the encoded data supplied from the content processing unit <NUM> to restore the data as output tactile data corresponding to the input tactile data as original tactile data and outputs the output tactile data to the content processing unit <NUM>. Note that the restoration method of the output tactile data from the encoded data executed by the decoding unit <NUM> will be described in detail later with reference to <FIG> and <FIG>.

The communication unit <NUM> is controlled by the control unit <NUM> and uses wired (or wireless (not illustrated)) communication to transmit and receive various data and programs to and from the transmission apparatus <NUM> through a communication network represented by a LAN (Local Area Network) or the like.

The storage unit <NUM> is controlled by the control unit <NUM> and includes an HDD (Hard Disk Drive), an SSD (Solid State Drive), a semiconductor memory, or the like. The storage unit <NUM> writes or reads various data including the content data and programs. In addition, the information of the base tactile signal group necessary for the decoding unit <NUM> to decode the encoded data is stored in the storage unit <NUM>.

Next, the encoding method and the decoding method of the tactile data will be described with reference to <FIG>.

In a case considered here, an input tactile signal based on input tactile data for vibrating one tactile presentation unit 53a is, for example, an input tactile signal with a vibration waveform as illustrated on the right side of <FIG>.

The encoding unit <NUM> performs frequency analysis, pattern matching, or the like to divide the input tactile signal as illustrated on the right side of <FIG> into a plurality of base tactile signals belonging to a base tactile signal group as illustrated on the left side of <FIG> to generate divided information.

On the left side of <FIG>, the input tactile signal is divided into three types of base tactile signals BTS1 to BTS3, and the offset (time code) of each is obtained.

More specifically, the waveform from time t1 to t51 on the right side of <FIG> is divided into base tactile signals BTS1-<NUM>, BTS2-<NUM>, and BTS3-<NUM> on the left side of <FIG>, and the waveform from time t22 to t52 on the right side of <FIG> is divided into base tactile signals BTS3-<NUM>, BTS3-<NUM>, BTS2-<NUM>, BTS1-<NUM>, and BTS1-<NUM> on the left side of <FIG>.

That is, as for the base tactile signal BTS1, the offset (time code) timing of each time from time t1 to t3 is the start timing of each of the base tactile signals BTS1-<NUM> to BTS1-<NUM>.

In addition, as for the base tactile signal BTS2, the offset (time code) timing of each of time t11 and time t12 is the start timing of each of the base tactile signals BTS2-<NUM> and BTS2-<NUM>.

Further, as for the base tactile signal BTS3, the start timing of each of the base tactile signals BTS3-<NUM> to BTS3-<NUM> is at the offset timing of each time from time t21 to t23.

The encoding unit <NUM> generates the encoded data based on the division results as obtained on the left side of <FIG>.

That is, the encoding unit <NUM> encodes the parameters regarding the base tactile signal including, for example, the information of the base tactile signal index for identifying the base tactile signal, the offset (time code) of each base tactile signal, and the length based on the division results as obtained in the left side of <FIG>.

Here, for the base tactile signal index, the encoding unit <NUM> uses a parameter of one of <NUM>, <NUM>, or <NUM> corresponding to the base tactile signals because there are three types of base tactile signals BTS1 to BTS3.

In addition, the offset (time code) is one of time t1 to t3, t11, t12, t21, t22, or t23 in the case on the left side of <FIG>, and the encoding unit <NUM> uses the time code indicating the corresponding offset as a parameter.

Further, the base tactile signals BTS1 to BTS3 have predetermined lengths T1 to T3, respectively, and the encoding unit <NUM> uses this information as a parameter.

As a result, on the left side of <FIG>, the encoding unit <NUM> encodes, for example, the base tactile signal BTS1-<NUM> as (base tactile signal index, time code indicating offset of base tactile signal, and length) = (<NUM>, t1, T1).

Similarly, the encoding unit <NUM> encodes the base tactile signals BTS1-<NUM> and BTS1-<NUM> as (base tactile signal index, time code indicating offset timing of base tactile signal, and length) = (<NUM>, t2, T1), (<NUM>, t3, T1), respectively.

In addition, the base tactile signals BTS2-<NUM> and BTS2-<NUM> are encoded as (base tactile signal index, time code indicating offset of base tactile signal, and length) = (<NUM>, t11, T2), (<NUM>, t12, T2), respectively.

Further, the base tactile signals BTS3-<NUM> to BTS3-<NUM> are encoded as (base tactile signal index, time code indicating offset of base tactile signal, and length) = (<NUM>, t21, T3), (<NUM>, t22, T3), (<NUM>, t23, T3), respectively.

Further, the encoding unit <NUM> adds and combines the encoding results obtained for the base tactile signals to generate the encoded data of the input tactile signal corresponding to the input tactile data. That is, the divided information of the input tactile signal as illustrated on the left side of <FIG> is used to combine the encoding results of the plurality of base tactile signals, such as, for example, (<NUM>, t1, T1), (<NUM>, t2, T1), (<NUM>, t3, T1), (<NUM>, t11, T2), (<NUM>, t12, T2), (<NUM>, t21, T3), and (<NUM>, t22, T3), and the encoded data of the input tactile signal is generated.

As a result of the series of processes, the input tactile signal corresponding to the input tactile data is converted into the encoded data.

On the other hand, the decoding unit <NUM> executes a process opposite the process of converting the input tactile data into the encoded data to convert the encoded data into the output tactile data.

That is, once the decoding unit <NUM> is supplied with the encoded data (<NUM>, t1, T1), (<NUM>, t2, T1), (<NUM>, t3, T1), (<NUM>, t11, T2), (<NUM>, t12, T2), (<NUM>, t21, T3), (<NUM>, t22, T3) generated as described above, the decoding unit <NUM> breaks up the plurality of pieces of combined encoded data to restore each base tactile signal.

That is, for example, (<NUM>, t1, T1) indicates that the base tactile signal index is <NUM>, the time code indicating the offset timing of the base tactile signal is t1, and the length is T1. Therefore, the decoding unit <NUM> reads the base tactile signal with the base tactile signal index of <NUM> in the base tactile signal group stored in the storage unit <NUM> and applies a process corresponding to offset = t1 to restore the base tactile signal BTS1-<NUM> on the left side of <FIG>.

Similarly, the decoding unit <NUM> restores the base tactile signals BTS1-<NUM>, BTS1-<NUM>, BTS2-<NUM>, BTS2-<NUM>, BTS3-<NUM>, BTS3-<NUM>, and BTS3-<NUM> as output tactile signals with signal waveforms as illustrated on the left side of <FIG> based on the encoded data (<NUM>, t2, T1), (<NUM>, t3, T1), (<NUM>, t11, T2), (<NUM>, t12, T2), (<NUM>, t21, T3), (<NUM>, t22, T3), and (<NUM>, t23, T3), respectively.

Further, the decoding unit <NUM> adds and combines the signal waveforms of the restoration results obtained as on the left side of <FIG> to restore the final signal waveforms of the output tactile signals as illustrated on the right side of <FIG> to generate the corresponding output tactile data. More specifically, the base tactile signals BTS1-<NUM>, BTS2-<NUM>, and BTS3-<NUM> illustrated on the left side of <FIG> are added to restore the output tactile signal with the waveform as illustrated from time t1 to t51 on the right side of <FIG>. In addition, the base tactile signals BTS3-<NUM>, BTS3-<NUM>, BTS2-<NUM>, BTS1-<NUM>, and BTS1-<NUM> illustrated on the left side of <FIG> are added to restore the output tactile signal with the waveform as illustrated from time t22 to t52 on the right side of <FIG>.

As a result of the encoding process and the decoding process, only the parameters of each of a predetermined number of base tactile signals need to be used to express the tactile data. Therefore, the tactile data can more efficiently be compressed compared to the existing tactile data in which the waveforms of the tactile signals based on the tactile data are digitalized point by associating the time and the waveform values in time series.

Therefore, even when the tactile data of multiple channels is transmitted and received, the tactile data can efficiently be transmitted and received, and the time required for the transmission and the reception can be shortened.

As a result, even when the tactile data of multiple channels is transmitted and received, the time required for the transmission and the reception can be reduced. Therefore, the generation of the delay regarding the tactile presentation can be suppressed even when the tactile data of multiple channels is transmitted and received, and the tactile presentation can be made in synchronization with the images and the audio content.

Although the base tactile signal index, the time code indicating the offset of the base tactile signal, and the length are used as the parameters of the encoded data in the description above, parameters for arranging the base tactile signal may be further added.

For example, changes in gain in a period of time defined by the length of the base tactile signal may be expressed by various envelopes, and a parameter including the type of envelope as an index may be set.

In addition, for example, certain changes in gain in the entire period of time defined by the length of the base tactile signal may be set as a parameter.

That is, the envelopes expressing the changes in gain in the period of time defined by the length of the base tactile signal are, for example, envelopes EV1 to EV4 as illustrated on the left side of <FIG>.

The envelope EV1 expresses that the gain is zero at time ts and te in the period of time from time ts to te determined by the length of the base tactile signal, and a certain gain is maintained for the base tactile signal between time ts and te.

The envelope EV2 expresses that the gain rises at a certain rate before intermediate time th, and the gain falls at a certain rate after intermediate time th in the period of time from time ts to te determined by the length of the base tactile signal.

The envelope EV3 expresses that the gain rises with a downward projecting waveform change before intermediate time th, and the gain falls with a downward projecting waveform change after intermediate time th in the period of time from time ts to te determined by the length of the base tactile signal.

The envelope EV4 expresses that the gain rises with an upward projecting waveform change before intermediate time th, and the gain falls with an upward projecting waveform change after intermediate time th in the period of time from time ts to te determined by the length of the base tactile signal.

The envelopes can be set in such a way to define fade-in and fade-out of the base tactile signals, and the proportions of the overlapping parts between the base tactile signals can appropriately be expressed.

For example, in a case where the envelope EV2 is applied to the base tactile signal BTS1, the gain in the entire time zone is <NUM>, and the time code as an offset is <NUM> as illustrated in the upper left part of <FIG>, the envelope EV2 changes the entire waveform of the base tactile signal BTS1 into a triangular shape projection in the up-down direction, and a base tactile signal BTS1' indicated by the waveform in the center part of <FIG> is obtained.

Further, in a case where, for example, the envelope EV4 is applied to the base tactile signal BTS2, the gain in the entire time zone is -6dB, and the time code as an offset is <NUM> as illustrated in the lower left part of <FIG>, the envelope EV4 changes the entire waveform of the base tactile signal BTS2 into a curve shape projection in the up-down direction, and the wave height of the waveform is reduced as a whole. Further, the waveform is offset by <NUM> in the time direction, and a base tactile signal BTS2' indicated by the waveform in the center part of <FIG> is obtained.

The base tactile signal BTS1' and the base tactile signal BTS2' in the center part of <FIG> are combined to obtain a combined waveform as illustrated on the right side of <FIG>.

Therefore, the parameters defining the gain and the envelope can be used to broaden the variation of the tactile signals expressed by combining the limited base tactile signals, and a more delicate tactile presentation can be made.

Note that in the encoding of the tactile data corresponding to the tactile signal in the following description, the parameters including the base tactile signal index, the time code (indicating the offset timing of the base tactile signal), the gain, the length, and the envelope index of each base tactile signal will be used as the encoded data.

Although the encoded data for the tactile data of one tactile presentation unit 53a has been described above, there is a plurality of tactile presentation units 53a in reality, and the encoded data is obtained in one channel for each of the plurality of tactile presentation units 53a.

That is, in the case where n tactile presentation units 53a-<NUM> to 53a-n are provided, the encoded data corresponding to n channels is obtained as illustrated in <FIG>.

In <FIG>, the encoded data of tactile presentation unit 1ch is described as the encoded data of the tactile presentation unit 53a-<NUM> in the top stage, and (base tactile signal index <NUM>, time code, gain, length, envelope index),. , (base tactile signal index <NUM>, time code, gain, length, envelope index),. , (base tactile signal index <NUM>, time code, gain, length, envelope index),. , and (base tactile signal index m, time code, gain, length, envelope index) are set.

In addition, the encoded data of tactile presentation unit 2ch is described as the encoded data of the tactile presentation unit 53a-<NUM> in the second stage, and (base tactile signal index <NUM>, time code, gain, length, envelope index),. , (base tactile signal index <NUM>, time code, gain, length, envelope index),. , (base tactile signal index <NUM>, time code, gain, length, envelope index),. , and (base tactile signal index m, time code, gain, length, envelope index) are set.

Further, the encoded data of tactile presentation unit 1ch is described as the encoded data of the tactile presentation unit 53a-n in the nth stage, and (base tactile signal index <NUM>, time code, gain, length, envelope index),. , (base tactile signal index <NUM>, time code, gain, length, envelope index),. , (base tactile signal index <NUM>, time code, gain, length, envelope index),. , and (base tactile signal index m, time code, gain, length, envelope index) are set.

That is, the encoded data of each of the base tactile signal indices <NUM> to m is set for each channel of each tactile presentation unit 53a.

Further, in a case where the parameters of the encoded data are the same as in another channel, information indicating that the parameters are the same as in another channel may be provided without describing the parameters.

That is, in a case where, for example, the time code of the base tactile signal index <NUM> in the tactile presentation unit 2ch is the same as the time code of the base tactile signal index <NUM>, (base tactile signal index <NUM>, 1ch, gain, 1ch (tx to ty), envelope index) may be described as illustrated in <FIG> to express that the information of the time code is the same as in the tactile presentation unit 1ch, the length of the base tactile signal is the same as in 1ch, and the same range is a range of time tx to ty.

Further, in a case where the gain of the base tactile signal index m is 5dB higher than the gain in the tactile presentation unit 1ch, (base tactile signal index m, time code, 1ch+<NUM>, length, envelope index) may be described as illustrated in <FIG> to express that the gain is 5dB higher than the gain in the tactile presentation unit 1ch.

Further, tactile presentation reference signs as standards may be set from the encoded data of the tactile presentation units 53a-<NUM> to 53a-n, and differences from the tactile presentation reference signs may be set as the encoded data of each channel.

The tactile presentation reference signs include, for example, the encoded data of (base tactile signal index <NUM>, time code, gain, length, envelope index),. , (base tactile signal index <NUM>, time code, gain, length, envelope index),. , (base tactile signal index <NUM>, time code, gain, length, envelope index),. , and (base tactile signal index m, time code, gain, length, envelope index) as illustrated in the top stage of <FIG>. The encoded data may include, for example, values, such as average values, minimum values, and maximum values of the parameters of the encoded data of the tactile presentation units 53a-<NUM> to 53a-n.

Further, the differences are set for the encoded data of the tactile presentation unit 53a of each channel.

That is, the encoded data of the tactile presentation unit 53a-<NUM> indicated in the tactile presentation unit 1ch includes encoded data of (base tactile signal index <NUM>, +<NUM>, -<NUM>, <NUM>, -<NUM>),. , (base tactile signal index <NUM>, -<NUM>, <NUM>, +<NUM>, -<NUM>),. , and (base tactile signal index m, +<NUM>, -<NUM>, <NUM>, +<NUM>) as illustrated in the second stage from the top of <FIG>.

In this case, the encoded data of the tactile presentation unit 53a-<NUM> indicated in the tactile presentation unit 1ch indicates that the time code of the encoded data of the base tactile signal index <NUM> is +<NUM> from the tactile presentation reference sign, the gain is -2dB from the tactile presentation reference sign, the length is the same as the tactile presentation reference sign, and the envelope index is -<NUM> from the tactile presentation reference sign.

In addition, the encoded data of the base tactile signal index <NUM> indicates that the time code is -<NUM> from the tactile presentation reference sign, the gain is the same as the tactile presentation reference sign, the length is +<NUM> from the tactile presentation reference sign, and the envelope index is -<NUM> from the tactile presentation reference sign.

Further, the encoded data of the base tactile signal index m indicates that the time code is +<NUM> from the tactile presentation reference sign, the gain is - 10dB from the tactile presentation reference sign, the length is the same as the tactile presentation reference sign, and the envelope index is +<NUM> from the tactile presentation reference sign.

In addition, as illustrated in the third stage from the top of <FIG>, the encoded data of the tactile presentation unit 53a-<NUM> indicated in the tactile presentation unit 2ch indicates that the encoded data is the same as the tactile presentation reference signs because all of the signs are <NUM>.

Further, the encoded data of the tactile presentation unit 53a-n indicated in the tactile presentation unit nch includes encoded data of (base tactile signal index <NUM>, +<NUM>, -<NUM>, <NUM>, +<NUM>),. , (base tactile signal index <NUM>, +<NUM>, <NUM>, +<NUM>, -<NUM>),. , and (base tactile signal index m, <NUM>, -<NUM>, +<NUM>, -<NUM>) as illustrated in the bottom stage of <FIG>.

In this case, the encoded data of the tactile presentation unit 53a-n indicated in the tactile presentation unit nch indicates that the time code of the encoded data of the base tactile signal index <NUM> is +<NUM> from the tactile presentation reference sign, the gain is -10dB from the tactile presentation reference sign, the length is +<NUM> from the tactile presentation reference sign, and the envelope index is +<NUM> from the tactile presentation reference sign.

In addition, the encoded data of the base tactile signal index <NUM> indicates that the time code is +<NUM> from the tactile presentation reference sign, the gain is the same as the tactile presentation reference sign, the length is +<NUM> from the tactile presentation reference sign, and the envelope index is -<NUM> from the tactile presentation reference sign.

Further, the encoded data of the base tactile signal index m indicates that the time code is the same as the tactile presentation reference sign, the gain is - 60dB from the tactile presentation reference sign, the length is +<NUM> from the tactile presentation reference sign, and the envelope index is -<NUM> from the tactile presentation reference sign.

In any case, the encoded data includes only the difference information. Therefore, the encoded data can more efficiently be compressed, and the communication speed of the encoded data can be improved. Further, in the case where all of the encoded data is the same as the tactile presentation reference signs as in the encoded data of the base tactile signal index <NUM> in <FIG>, one sign may be used to express that all of the encoded data of the base tactile signal index <NUM> is the same as the tactile presentation reference signs.

Next, a reproduction process of the content executed by the content reproduction system of <FIG> will be described with reference to a flow chart of <FIG>.

In step S11, the content processing unit <NUM> reads the content data including the image data, the audio data, and the tactile data (input tactile data) stored in the storage unit <NUM>.

In step S12, the content processing unit <NUM> extracts the image data, the audio data, and the tactile data (input tactile data) from the read content data.

In step S13, the tactile data processing unit <NUM> executes a tactile data encoding process to convert the tactile data (input tactile data) into encoded data.

Here, the tactile data encoding process will be described with reference to a flow chart of <FIG>.

In step S51, the tactile data processing unit <NUM> in the control unit <NUM> of the transmission apparatus <NUM> reads the tactile data (input tactile data) extracted by the content processing unit <NUM> from the content data.

In step S52, the encoding unit <NUM> generates the tactile signal (input tactile signal) based on the tactile data (input tactile data). Here, the tactile signal has, for example, the signal waveform as illustrated on the right side of <FIG>.

In step S53, the encoding unit <NUM> applies frequency analysis or pattern matching to divide the tactile signal (input tactile signal) with the signal waveform into the base tactile signal group and generates the divided information. That is, in the case of, for example, the tactile signal with the signal waveform as illustrated on the right side of <FIG>, the encoding unit <NUM> generates the divided information including the base tactile signal group as illustrated on the left side of <FIG>.

In step S54, the encoding unit <NUM> sets parameters for each base tactile signal group based on the divided information. More specifically, the encoding unit <NUM> sets the parameters corresponding to the base tactile signal index, the time code, the gain, the length, and the envelope index for each base tactile signal group based on the divided information as illustrated on the left side of <FIG>.

In step S55, the encoding unit <NUM> encodes the parameters set for each base tactile signal group that are divided information and outputs, as encoded data, the combined information obtained by combining the parameters.

Here, the flow chart of <FIG> will be further described.

In step S14, the content processing unit <NUM> controls the communication unit <NUM> to transmit the content data including the image data and the audio data in addition to the encoded data to the reception apparatus <NUM>.

In step S31, the content processing unit <NUM> in the control unit <NUM> of the reception apparatus <NUM> controls the communication unit <NUM> to receive the content data including the image data and the audio data in addition to the encoded data transmitted from the transmission apparatus <NUM>.

In step S32, the tactile data processing unit <NUM> executes a tactile data decoding process to decode the encoded data in the content data received by the content processing unit <NUM> into tactile data (output tactile data).

Here, the tactile data decoding process will be described with reference to a flow chart of <FIG>.

In step S71, the tactile data processing unit <NUM> reads the encoded data in the content data received by the content processing unit <NUM>.

In step S72, the decoding unit <NUM> reads the parameters set for each base tactile encoding group based on the encoded data.

In step S73, the decoding unit <NUM> restores the divided information set for each base tactile encoding group based on the read parameters. That is, the decoding unit <NUM> restores the divided information including the base tactile signal group as illustrated on the left side of <FIG> based on the parameters of each of the base tactile signal index, the time code, the gain, the length, and the envelope index. That is, the decoding unit <NUM> reads the base tactile signals corresponding to the base tactile signal index from the storage unit <NUM> and restores the divided information of each base tactile signal by applying a process based on the parameters of each of the time code, the gain, the length, and the envelope index.

In step S74, the decoding unit <NUM> combines the divided information to restore the tactile signal (as output tactile signal). That is, the decoding unit <NUM> restores the tactile signal (as output tactile signal) as illustrated on the right side of <FIG> based on the divided information including the base tactile signal group as illustrated on the left side of <FIG>.

In step S75, the decoding unit <NUM> generates the tactile data (output tactile data) based on the tactile signal (output tactile signal).

In step S33, the content processing unit <NUM> outputs the image data and the audio data to the display apparatus <NUM> and causes the display apparatus <NUM> to display the images of the content and output the sound to thereby reproduce the content.

In step S34, the content processing unit <NUM> supplies the decoded tactile data (output tactile data) to the tactile presentation apparatus <NUM> and causes each of the tactile presentation units 53a to vibrate to reproduce the tactile data (make a tactile presentation).

Note that in this case, the content processing unit <NUM> uses timing control information independent from each other to manage and execute the timing of the process of supplying the transmitted image data and audio data to the display apparatus <NUM> to cause the display apparatus <NUM> to reproduce the content and the process of supplying the tactile data (output tactile data) to the tactile presentation apparatus <NUM> to cause the tactile presentation apparatus <NUM> to make the tactile presentation.

In addition, whether or not the end is instructed is determined in steps S15 and S35. In a case where the end is not instructed, the process returns to steps S11 and S31, and the subsequent processes are repeated.

Further, in a case where the end is instructed in steps S15 and S35, the reproduction process ends.

As a result of the process, the tactile data (input tactile data) is converted into the encoded data, and the encoded data is compressed and transmitted. After the encoded data is received, the tactile data (output tactile data) is restored. Therefore, the tactile data can efficiently be transmitted and received, and the delay in the tactile presentation by the tactile presentation unit 53a can be suppressed in the reproduction of the content.

As a result, even when the tactile data of multiple channels involving a plurality of tactile presentation units 53a is transmitted and received, the tactile data can efficiently be transmitted and received. Therefore, the increase in the time required for the transmission and the reception can be suppressed, and the delay in the tactile presentation can be suppressed.

In addition, although there is one tactile presentation unit 53a in the case of the process described above, the process of steps S11 to S13 may be executed for each channel in a case where there is a plurality of tactile presentation units 53a so that the encoded data corresponding to the tactile data of multiple channels is necessary. The encoded data as processing results may be set as, for example, integrated encoded data corresponding to multiple channels as described with reference to <FIG>, and the encoded data may be transmitted to the reception apparatus <NUM> in step S14.

In this case, the encoded data corresponding to multiple channels may be decoded and converted into tactile data in step S32, and the plurality of tactile presentation units 53a may make a tactile presentation in step S34.

As a result, the same information as the information of another channel can be eliminated as described with reference to <FIG> and <FIG>. Therefore, the tactile data can efficiently be transmitted and received, and the delay in the tactile presentation can more efficiently be suppressed.

In the example described above, the encoded data including the encoded tactile data (input tactile data) in addition to the image data and the audio data is transmitted from the transmission apparatus <NUM> to the reception apparatus <NUM>. After the encoded data is received, the tactile data is restored (as output tactile data) from the encoded data. The timing control information used for reproducing the image data and the audio data and the timing control information used for reproducing the tactile data (output tactile data) are independent from each other.

However, the reception apparatus <NUM> may store the image data and the audio data in the storage unit <NUM> or the removable storage medium <NUM> in advance, and only the encoded data including the encoded tactile data (input tactile data) may be transmitted from the transmission apparatus <NUM> to the reception apparatus <NUM>.

In such a way, the information transmitted and received to and from the transmission apparatus <NUM> and the reception apparatus <NUM> is only the encoded data converted from the tactile data (input tactile data). Therefore, the processing time required for the transmission and the reception can be shortened, and the generation of the delay in the tactile presentation caused by the increase in the transmission and reception time of the tactile data can be suppressed.

Next, a reproduction process of the content in the first modification will be described with reference to a flow chart of <FIG>. Note that the processes of steps S101 to S103, S106, and S135 in <FIG> are similar to the processes of steps S11 to S13, S15, and S35 described with reference to a flow chart of <FIG>, and the description will not be repeated.

That is, the image data, the audio data, and the encoded data are prepared in steps S101 to S103.

Further, in step S104, the content processing unit <NUM> provides an index corresponding to the reproduction timing of the image data and the audio data to the generated encoded data.

In step S105, the content processing unit <NUM> controls the communication unit <NUM> to transmit only the encoded data provided with the index to the reception apparatus <NUM>.

In step S131, the content processing unit <NUM> in the control unit <NUM> of the reception apparatus <NUM> controls the communication unit <NUM> to receive the encoded data provided with the index transmitted from the transmission apparatus <NUM>.

In step S132, the tactile data processing unit <NUM> executes the tactile data decoding process to decode the encoded data in the content data received by the content processing unit <NUM> into the tactile data (output tactile data). In this case, the tactile data processing unit <NUM> provides the index provided to the encoded data before decoding to the tactile data after decoding (output tactile data). Note that the tactile data decoding process is similar to the process described with reference to the flow chart of <FIG>, and the process will not be described.

In step S133, the content processing unit <NUM> reads the image data and the audio data stored in the storage unit <NUM> or the removable storage medium <NUM> and outputs the image data and the audio data to the display apparatus <NUM>. The content processing unit <NUM> causes the display apparatus <NUM> to display the images of the content and output the sound.

In step S134, the content processing unit <NUM> supplies, to the tactile presentation apparatus <NUM>, the tactile data (output tactile data) corresponding to the index indicating the reproduction timing of the image data and the audio data reproduced by the display apparatus <NUM> and causes the tactile presentation units 53a to make a tactile presentation based on the tactile data (output tactile data).

That is, in this case, only the encoded data provided with the index corresponding to the reproduction timing of the image data and the audio data is transmitted and received. Therefore, the transmission and reception time can be shortened, and the delay in the timing of the tactile presentation by the tactile presentation units 53a from the timing of the output of the image data and the audio data from the display apparatus <NUM> can be prevented.

In addition, the tactile presentation units 53a make the tactile presentation based on the tactile data (output tactile data) provided with the index corresponding to the reproduction timing of the image data and the audio data. Therefore, the timing of the reproduction of the image and the audio can be synchronized with the timing of the tactile presentation.

Further, even when the tactile presentation units 53a transmit and receive the tactile data of multiple channels, the increase in the transmission and reception time can be suppressed. Therefore, the delay in the tactile presentation can be suppressed.

In the example described above, only the encoded data corresponding to the tactile data provided with the index corresponding to the image data and the audio data is transmitted from the transmission apparatus <NUM> to the reception apparatus <NUM>, and the reception apparatus <NUM> reproduces the image data and the audio data stored in the reception apparatus <NUM> and synchronously reproduces the tactile data provided with the corresponding index.

However, the encoded data may be stored in transfer data in a format (multimedia container format) in which the image data, the audio data, and the encoded data (including encoded input tactile data) are integrated in an identifiable state, and a time code for synchronizing the reproduction of the tactile data (output tactile data) decoded from the encoded data with the reproduction of the image data and the audio data may be recorded as meta information in the transfer data. In this case, the transfer data in the format in which the image data, the audio data, and the encoded data corresponding to the encoded data are integrated in an identifiable state is transmitted from the transmission apparatus <NUM> to the reception apparatus <NUM>. Once the reception apparatus <NUM> decodes the tactile data (output tactile data) from the encoded data included in the transfer data, the reception apparatus <NUM> reads the time code recorded as meta information and provides the tactile presentation based on the tactile data (output tactile data) in synchronization with the reproduction of the image data and the audio data.

Next, a reproduction process of the content in the second modification will be described with reference to a flow chart of <FIG>. Note that the processes of steps S151 to <NUM>, S156, and S175 in <FIG> are similar to the processes of steps S11 to S13, S15, and S35 described with reference to the flow chart of <FIG>, and the description will not be repeated.

That is, the image data, the audio data, and the encoded data are prepared in steps S151 to S153.

Further, in step S154, the content processing unit <NUM> stores the generated encoded data, the image data, and the audio data in the transfer data in the format in which the data is integrated in an identifiable state.

In step S155, the content processing unit <NUM> embeds, as meta information of the transfer data, the time code for reproducing the tactile data (output tactile data) decoded from the encoded data in synchronization with the image data and the audio data.

In step S156, the content processing unit <NUM> controls the communication unit <NUM> to transmit the transfer data to the reception apparatus <NUM>.

In step S171, the content processing unit <NUM> in the control unit <NUM> of the reception apparatus <NUM> controls the communication unit <NUM> to receive the transfer data transmitted from the transmission apparatus <NUM>. As described above, the encoded data, the image data, and the audio data are stored in the transfer data in an identifiable state.

In step S172, the tactile data processing unit <NUM> executes the tactile data decoding process to extract the encoded data from the transfer data received by the content processing unit <NUM> to decode the encoded data into the tactile data (output tactile data). Note that the tactile data decoding process is similar to the process described with reference to the flow chart of <FIG>, and the process will not be described.

In step S173, the content processing unit <NUM> reads the image data and the audio data and outputs the image data and the audio data to the display apparatus <NUM>. The content processing unit <NUM> causes the display apparatus <NUM> to display the images of the content and output the sound.

In step S174, the content processing unit <NUM> reads the time code embedded as meta information of the transfer data and supplies the tactile data (output tactile data) to the tactile presentation apparatus <NUM> in synchronization with the image data and the audio data reproduced by the display apparatus <NUM> based on the time code to cause the tactile presentation units 53a to make the tactile presentation.

That is, in this case, the content processing unit <NUM> supplies the tactile data (output tactile data) to the tactile presentation apparatus <NUM> to cause the tactile presentation apparatus <NUM> to make the tactile presentation in synchronization with the image data and the audio data supplied to and reproduced by the display apparatus <NUM> based on the time code embedded as meta information. Therefore, the reproduction timing of the image data and the audio data in the display apparatus <NUM> and the timing of the tactile presentation in the tactile presentation units 53a based on the tactile data (output tactile data) can be synchronized. As a result, the delay in the timing of the tactile presentation by the tactile presentation units 53a from the timing of the output of the image data and the audio data from the display apparatus <NUM> is prevented.

As a result of the process, the tactile data is converted into the encoded data, and the encoded data is compressed, transmitted, and received. Therefore, the tactile data can efficiently be transmitted and received, and the communication time required for transmitting and receiving the tactile data can be shortened. This can prevent the delay in the tactile presentation by the tactile presentation units 53a in reproducing the content due to the increase in the transmission and reception time of the tactile data of multiple channels.

In the example described above, the time code for reproducing the tactile data in synchronization with the image data and the audio data is embedded as meta information in the transfer data in which the encoded data including the encoded tactile data, the image data, and the audio data are integrated in an identifiable state. The transmission apparatus <NUM> transmits the transfer data to the reception apparatus <NUM>. The reception apparatus <NUM> reproduces the image data and the audio data and reproduces the tactile data (output tactile data) restored from the encoded data in synchronization with the image data and the audio data based on the time coded embedded as meta information.

However, only the tactile data that can be decoded by the reception apparatus <NUM> may be converted into the encoded data. As for the tactile data that cannot be decoded, raw tactile data may be transmitted from the transmission apparatus <NUM> to the reception apparatus <NUM> along with the image data and the audio data.

Next, a reproduction process of the content in the third modification will be described with reference to a flow chart of <FIG>. Note that the processes of steps S191, <NUM>, S194, S195, S197, and S203 to S206 in <FIG> are similar to the processes of steps S11 to S15 and steps S32 to S35 described with reference to the flow chart of <FIG>, and the description will not be repeated.

That is, the image data, the audio data, and the tactile data (input tactile data) are read in steps S191 and S192.

In step S193, when the tactile data processing unit <NUM> encodes the tactile data (input tactile data), the tactile data processing unit <NUM> determines whether or not the tactile data is undecodable in the reception apparatus <NUM>.

More specifically, the tactile data processing unit <NUM> may, for example, convert the tactile data (input tactile data) once into the encoded data in the tactile data encoding process and then decode the encoded data into the tactile data (output tactile data) in the tactile data decoding process. The tactile data processing unit <NUM> may compare the output tactile data as a decoding result and the input tactile data to determine whether the tactile data is undecodable based on the degree of similarity.

In a case where it is determined that the tactile data is decodable instead of undecodable in step S193, the process proceeds to step S194, and the tactile data (input tactile data) is converted into the encoded data in the tactile data encoding process. In step S195, the encoded data is transmitted to the reception apparatus <NUM> along with the image data and the audio data.

On the other hand, in a case where it is assumed that the tactile data is undecodable in step S193, the process proceeds to step S196.

In step S196, the content processing unit <NUM> controls the communication unit <NUM> to transmit the raw tactile data (input tactile data), the image data, and the audio data to the reception apparatus <NUM>.

In step S201, the content processing unit <NUM> in the control unit <NUM> of the reception apparatus <NUM> controls the communication unit <NUM> to receive the raw tactile data (input tactile data) or the encoded data, the image data, and the audio data transmitted from the transmission apparatus <NUM>.

In step S202, the tactile data processing unit <NUM> determines whether or not the encoded data is received along with the image data and the audio data. For example, in a case where the encoded data is received along with the image data and the audio data, the process proceeds to step S203.

Further, the encoded data is decoded into the tactile data (output tactile data) in the tactile data decoding process in step S203. In step S204, the display apparatus <NUM> outputs the images and the sound based on the image data and the audio data. In step S205, the tactile presentation units 53a make the tactile presentation based on the tactile data (output tactile data).

Further, in a case where the raw tactile data (input tactile data), instead of the encoded data, is received along with the image data and the audio data in step S202, the process of step S203 is skipped. In step S204, the images and the sound are output. In step S205, the tactile presentation units 53a make the tactile presentation based on the raw tactile data (input tactile data).

That is, in this case, the content processing unit <NUM> converts the encoded data into the tactile data (output tactile data) to make the tactile presentation only for the tactile data (input tactile data) that can appropriately be decoded. For the tactile data (input tactile data) that cannot be decoded, the content processing unit <NUM> uses the raw tactile data (input tactile data) to make the tactile presentation. Therefore, the tactile data can appropriately be reflected to make the tactile presentation.

As a result of the process, the tactile data is converted into the encoded data, and the encoded data is compressed, transmitted, and received. In this way, the tactile data can efficiently be transmitted and received. Therefore, the communication time required for transmitting and receiving the tactile data can be shortened. Even when the tactile data of multiple channels is transmitted and received in the reproduction of the content, an appropriate tactile presentation can be realized while suppressing the delay in the tactile presentation by the tactile presentation units 53a due to the increase in the transmission and reception time.

Note that in the tactile data encoding process and the tactile data decoding process, the number of used base tactile signals may be changed according to the communication band that can be used in the communication between the transmission apparatus <NUM> and the reception apparatus <NUM>.

Further, in the reception apparatus <NUM>, the tactile data as a decoding result of the encoded data may be stored in the storage unit <NUM>. Further, in a case of transmitting tactile data similar to the stored tactile data, the transmission apparatus <NUM> may transmit only tactile data specification information for specifying the similarity to certain tactile data. In this case, the reception apparatus <NUM> may use the tactile data specified based on the tactile data specification information in the tactile data stored in the storage unit <NUM>.

Further, the tactile data may be transmitted and received by switching in real time whether to transmit the encoded data, transmit the tactile data specification information for specifying the already stored tactile data, or transmit the raw tactile data according to the communication conditions, or the transmission methods may be combined.

In addition, part of the tactile data can automatically be generated from the audio data, and therefore, only the difference from the tactile data that can automatically be generated may be transmitted and received as tactile data.

Further, whether or not the tactile data encoding process and the tactile data decoding process can be executed may be determined according to the processing load or the remaining battery of the transmission apparatus <NUM> and the reception apparatus <NUM>, and the case of transmitting the encoded data and the case of transmitting the raw tactile data may be switched.

In addition, a time stamp on the content for designating the timing of reproducing the tactile data may be added to each base tactile signal, and the tactile presentation may be made according to the time stamp.

Further, the tactile data may be transmitted and received after multiplexing the tactile data with the content data, or the content data and the tactile data may be transmitted as different data.

In addition, flag data indicating whether or not the tactile data is added or indicating the format of the tactile data may be added to the header of the content data. In such a way, the reception apparatus <NUM> can determine whether or not there is tactile data or determine the format of the tactile data, and the processes can be distinguished and executed.

In the example described above, the reception apparatus <NUM> causes the display apparatus <NUM> to display the images and output the sound based on the image data and the audio data included in the content data supplied from the transmission apparatus <NUM> and further causes the tactile presentation units 53a of the tactile presentation apparatus <NUM> to make the tactile presentation based on the encoded data corresponding to the tactile data. However, a reception apparatus that causes the display apparatus <NUM> to display the images and output the sound based on the image data and the audio data and a reception apparatus that causes the tactile presentation units 53a of the tactile presentation apparatus <NUM> to make the tactile presentation based on the encoded data corresponding to the tactile data may be separately provided.

In addition, other than the transmission apparatus <NUM>, an apparatus with a function of converting the tactile data in the content data into the encoded data may be separately provided, and the transmission apparatus <NUM> may supply the tactile data to the separately provided apparatus and acquire the converted encoded data.

Further, other than the reception apparatus <NUM>, an apparatus with a function of decoding the encoded data into the tactile data may be provided, and the reception apparatus <NUM> may supply the encoded data to the separately provided apparatus and use the tactile data decoded from the encoded data to cause the tactile presentation units 53a of the tactile presentation apparatus <NUM> to make the tactile presentation.

In the examples described above, the frequency analysis or the pattern matching is used to divide the tactile signal (input tactile signal) into the base tactile signal group based on the tactile data (input tactile data), and the divided information is obtained. The parameters set for each base tactile signal based on the divided information are used to encode the divided information and combine the divided information, and the generated combined signal is set to generate the encoded data. The encoded data is decoded to restore the tactile data (output tactile data).

However, the process of dividing the tactile data (input tactile data) into the base tactile signal group to generate the divided information may be a process using an estimator generated based on machine learning.

Here, the tactile data encoding process using machine learning will be described.

In the machine learning, for example, content data <NUM> including image data and audio data and a corresponding tactile signal (input tactile signal) <NUM> are used as illustrated in <FIG>. Note that the tactile signal <NUM> here is a signal input by the user in association with the content data <NUM>.

Next, feature data <NUM> including a multi-dimensional feature vector including auditory features extracted from the image data and a multi-dimensional feature vector extracted from visual features extracted from the audio data is generated based on analysis results of audio analysis, dynamic analysis, texture analysis, and the like of the image data and the audio data of the content data <NUM>.

In addition, the tactile signal (input tactile signal) <NUM> is divided based on the base tactile signal group to generate divided information, and divided information <NUM> as teacher data is generated.

Further, an estimator (estimator <NUM> of <FIG>) that uses the feature data <NUM> of the content data <NUM> to estimate the divided information is generated by machine learning using the feature data <NUM> and the divided information <NUM> as teacher data.

The estimator (estimator <NUM> of <FIG>) estimates the divided signal of the tactile signal (input tactile signal) through the series of machine learning based on the feature data <NUM>.

As a result, as illustrated in <FIG>, a tactile data encoding process using the estimator <NUM> generated by machine learning is executed to generate feature data <NUM> including a multi-dimensional feature vector from which the auditory features of the image data are extracted and a multi-dimensional feature vector extracted from the visual features of the audio data based on analysis results of audio analysis, dynamic analysis, texture analysis, and the like of the image data and the audio data included in content data <NUM>.

Further, the estimator <NUM> generated by machine learning estimates divided information based on the feature data <NUM> and generates encoded data based on each parameter of the base tactile signal group in the estimated divided information.

Next, a configuration example of a learning apparatus that uses machine learning to generate the estimator <NUM> of <FIG> configured to estimate the divided information based on the feature data of the content will be described with reference to <FIG>.

A learning apparatus <NUM> includes a control unit <NUM>, a communication unit <NUM>, a storage unit <NUM>, an input unit <NUM>, an output unit <NUM>, a drive <NUM>, and a removable storage medium <NUM> that are connected to each other through a bus <NUM> so that data and programs can be transmitted and received.

The control unit <NUM> includes a processor and a memory and controls the entire operation of the learning apparatus <NUM>. The control unit <NUM> includes a machine learning unit <NUM> that causes the estimator <NUM> to learn through machine learning.

The machine learning unit <NUM> includes a feature data extraction unit <NUM> and a teacher data generation unit <NUM>. As described with reference to <FIG>, the machine learning unit <NUM> acquires the content data <NUM> and the tactile signal (input tactile signal) <NUM> on the basis of the corresponding tactile data and causes the estimator <NUM> to learn through, for example, a neural network, a Gaussian mixture model, or the like.

More specifically, the feature data extraction unit <NUM> extracts the feature data <NUM> including the multi-dimensional feature vector including the auditory features extracted from the image data and the multi-dimensional feature vector extracted from the visual features extracted from the audio data based on the analysis results of the audio analysis, the dynamic analysis, the texture analysis, and the like of the image data and the audio data of the content data <NUM>.

The teacher data generation unit <NUM> divides the tactile signal (input tactile signal) <NUM> on the basis of the tactile data corresponding to the content data <NUM> input through the input unit <NUM> to generate the divided information <NUM> including the base tactile signal group.

The machine learning unit <NUM> generates the estimator <NUM> that estimates the divided information based on the feature data <NUM> through machine learning using the feature data <NUM> and the divided information <NUM> as teacher data.

The communication unit <NUM> is controlled by the control unit <NUM> and transmits and receives various data and programs through a communication network represented by a LAN (Local Area Network) or the like in wired (or wireless (not illustrated)) communication.

Next, a configuration example of a transmission apparatus that realizes the tactile data encoding process using machine learning will be described with reference to <FIG>.

Note that the components of the transmission apparatus <NUM> in <FIG> with the same functions as in the components of the transmission apparatus <NUM> in <FIG> are provided with the same signs, and the description will be appropriately skipped.

That is, the transmission apparatus <NUM> of <FIG> is different from the transmission apparatus <NUM> of <FIG> in that an encoding unit <NUM> is provided in place of the encoding unit <NUM>.

The encoding unit <NUM> includes a feature data extraction unit <NUM> and an estimator <NUM> and generates the encoded data of the tactile data based on the image data and the audio data included in the content data.

More specifically, the feature data extraction unit <NUM> extracts the feature data from the image data and the audio data.

The estimator <NUM> has a configuration corresponding to the estimator <NUM> of <FIG>. The estimator <NUM> learns based on the machine learning and estimates the divided information corresponding to the tactile data (input tactile data) based on the feature data. The encoding unit <NUM> generates the encoded data corresponding to the tactile data (input tactile data) based on the parameters of each base tactile signal of the estimated divided information.

Note that the configuration of the reception apparatus <NUM> is also similar to the case in <FIG>, and the reproduction process is similar to the process described above. Therefore, the description will not be repeated.

Next, a learning process executed by the learning apparatus <NUM> of <FIG> will be described with reference to a flow chart of <FIG>.

In step S211, the feature data extraction unit <NUM> extracts the feature data <NUM> from the image data and the audio data of the content data <NUM> input through the input unit <NUM>.

In step S212, the teacher data generation unit <NUM> divides the tactile signal (input tactile signal) <NUM> on the basis of the tactile data corresponding to the content data <NUM> input through the input unit <NUM> to generate the divided information <NUM> including the base tactile signal group and generates the divided information <NUM> as teacher data.

In step S213, the machine learning unit <NUM> causes the estimator <NUM> to learn through machine learning using the feature data <NUM> including the tactile data and the divided information <NUM> as teacher data.

The process can cause the estimator <NUM> that estimates the divided information of the tactile signal (input tactile signal) to learn based on the feature data extracted from the image data and the audio data of the content data in machine learning.

Next, a tactile data encoding process using the estimator <NUM> (estimator <NUM> of <FIG>) generated through machine learning of the transmission apparatus <NUM> in <FIG> will be described with reference to <FIG>.

In step S231, the feature data extraction unit <NUM> of the encoding unit <NUM> extracts the feature data from the image data and the audio data of the content data.

In step S232, the estimator <NUM> estimates the divided information of the tactile signal (input tactile signal) corresponding to the feature data.

In step S233, the encoding unit <NUM> sets the parameters of each base tactile signal based on the divided information estimated by the estimator <NUM>.

In step S234, the encoding unit <NUM> generates the encoded data corresponding to the tactile signal based on the parameters of each base tactile signal.

As a result of the process, the estimator <NUM> generated by machine learning estimates the divided information based on the feature data of the image data and the audio data and estimates the encoded data corresponding to the tactile data (input tactile data). Therefore, the tactile data (input tactile data) can more quickly be converted into the encoded data. Note that the tactile data decoding process is similar to the process described with reference to the flow chart of <FIG>, and the description will not be repeated. In addition, although the feature data is extracted from the image data and the audio data in the learning process and the tactile data encoding process in the example described above, the feature data may also be extracted from the tactile data (input tactile data) in addition to the image data and the audio data. Further, the feature data can be increased so that the estimator can learn with a higher precision.

As described above, according to the present disclosure, the increase in the transmission and reception time can be suppressed in any case even when a plurality of tactile presentation units 53a transmits and receives tactile data of multiple channels, and therefore, the delay in the tactile presentation can be suppressed.

Incidentally, the series of processes described above can be executed by hardware or can be executed by software. In the case where the series of processes are executed by software, a program included in the software is installed from a recording medium on a computer incorporated into dedicated hardware or on, for example, a general-purpose computer or the like that can execute various functions by installing various programs.

<FIG> illustrates a configuration example of the general-purpose computer. The personal computer includes a CPU (Central Processing Unit) <NUM>. An input-output interface <NUM> is connected to the CPU <NUM> through a bus <NUM>. A ROM (Read Only Memory) <NUM> and a RAM (Random Access Memory) <NUM> are connected to the bus <NUM>.

An input unit <NUM> including an input device, such as a keyboard and a mouse, for the user to input an operation command, an output unit <NUM> that outputs a processing operation screen or an image of a processing result to a display device, a storage unit <NUM> including a hard disk drive or the like that stores programs and various data, and a communication unit <NUM> including a LAN (Local Area Network) adapter or the like and configured to execute a communication process through a network represented by the Internet are connected to the input-output interface <NUM>. In addition, a drive <NUM> that reads and writes data to and from a removable storage medium <NUM>, such as a magnetic disk (including a flexible disk), an optical disk (including a CD-ROM (Compact Disc-Read Only Memory) and a DVD (Digital Versatile Disc)), a magneto-optical disk (including an MD (Mini Disc)), and a semiconductor memory, is connected.

The CPU <NUM> executes various processes according to a program stored in the ROM <NUM> or a program read from the removable storage medium <NUM>, such as a magnetic disk, an optical disk, a magneto-optical disk, and a semiconductor memory, installed on the storage unit <NUM>, and loaded from the storage unit <NUM> to the RAM <NUM>. Data and the like necessary for the CPU <NUM> to execute various processes are also appropriately stored in the RAM <NUM>.

In the computer configured in this way, the CPU <NUM> loads, for example, the program stored in the storage unit <NUM> to the RAM <NUM> through the input-output interface <NUM> and the bus <NUM> and executes the program to thereby execute the series of processes.

The program executed by the computer (CPU <NUM>) can be provided by, for example, recording the program in the removable storage medium <NUM> as a package medium or the like. In addition, the program can be provided through wired or wireless transmission medium, such as a local area network, the Internet, and digital satellite broadcasting.

In the computer, the removable storage medium <NUM> can be mounted on the drive <NUM> to install the program on the storage unit <NUM> through the input-output interface <NUM>. In addition, the communication unit <NUM> can receive the program through a wired or wireless transmission medium to install the program on the storage unit <NUM>. Further, the program can be installed in advance on the ROM <NUM> or the storage unit <NUM>.

Note that the program executed by the computer may be a program in which the processes are executed in chronological order described in the present specification, or the program may be a program for executing the processes in parallel or at necessary timing such as when the processes are invoked.

Note that the CPU <NUM> in <FIG> realizes the functions of the control unit <NUM> in <FIG> and <FIG>, the control unit <NUM> in <FIG>, and the control unit <NUM> in <FIG>.

In addition, the system in the present specification denotes a set of a plurality of constituent elements (apparatuses, modules (components), and the like), and whether or not all of the constituent elements are in the same housing does not matter. Therefore, a plurality of apparatuses stored in separate housings and connected through a network and one apparatus storing a plurality of modules in one housing are both systems.

Note that the embodiments of the present disclosure are not limited to the embodiments described above, and various changes can be made without departing from the scope of the present disclosure.

For example, the present disclosure can be provided as, for example, cloud computing in which a plurality of apparatuses shares one function and cooperates to execute a process through a network.

In addition, one apparatus can execute each step described in the flow charts, or a plurality of apparatuses can take charge and execute each step.

Claim 1:
An information processing apparatus (<NUM>) comprising:
an encoding unit (<NUM>) configured to
encode an input tactile signal into encoded data including parameters defining a plurality of base tactile signals in a base tactile signal group included in the input tactile signal, the input tactile signal having a vibration waveform for a tactile presentation unit (<NUM>; 13a),
wherein the base tactile signals included in the base tactile signal group include tactile signals orthogonal to each other,
wherein the parameters defining each of the base tactile signals include a respective index identifying the respective base tactile signal, a respective time code defining a respective offset of reproduction timing of the respective base tactile signal, a respective gain in entire timing of reproducing the respective base tactile signal, a respective length of tactile presentation based on the respective base tactile signal, and a respective index identifying an envelope indicating a change in gain in reproducing the respective base tactile signal; and
a transmission unit (<NUM>) configured to
transmit the encoded data including the parameters to a reception apparatus (<NUM>) that causes the tactile presentation unit (13a; 53a) to make a tactile presentation using an output tactile signal restored on a basis of the encoded data to make a tactile presentation, the output tactile signal being an approximate reconstruction of the input tactile signal based on the encoded data.