Patent Description:
In recent years, with the development of miniaturization, simplification, and the like of various types of motion sensors and biological information sensors, it has become easier to acquire various types of sensing data with these sensors, and the sensing data have been used, for example, for assistance of user's learning of a performance (playing of a musical instrument, a sport, and the like). For example, as a technology for evaluating a performance (playing of a musical instrument) and assisting in the performance, a technology disclosed in PTL <NUM> below can be cited. In such learning assistance, data that can be used in the learning assistance is obtained by acquiring various pieces of sensing data from a plurality of sensors and analyzing the plurality of acquired sensing data. PTL <NUM> discloses a signal processing unit provided in a mounting member worn on the hand of a user, successively acquiring the output values of a plurality of strain sensors that measure a change of shape of the user's finger from a data glove equipped with the strain sensors and configured to correct an output value of a sensor without making a user aware of a timing at which correction of the output value is to be executed.

Meanwhile, in the case of intending to acquire a plurality of pieces sensing data using a plurality of different types of sensors, the sensing data sometimes differs in form. For example, sensing data from a biological information sensor is normally in an analog form. In such a case, the analog form of the sensing data from the biological information sensor is converted into a digital form to make the sensing data in the form similar to that of sensing data from other sensors, and the sensing data from the biological information sensor in the digital form is then analyzed together with the sensing data from the other sensors. Furthermore, to acquire data useful for the assistance of user's learning of the performance, it is required to accurately synchronize a plurality of pieces of sensing data even in a mixture of forms as described above.

Therefore, the present disclosure proposes a novel and improved information processing apparatus capable of accurately synchronizing and handling sensing data in a mixture of forms and available for assisting in learning of a performance.

The present disclosure provides an information processing apparatus as defined in appended claim <NUM>.

As described so far, according to the present disclosure, it is possible to provide an information processing apparatus capable of accurately synchronizing and handling sensing data in a mixture of forms and available for assisting in learning of a performance.

It is noted that effects are not always limited to the above effect, the present disclosure may exhibit any of the effects described in the present specification or other effects that can be conceived from the present specification in addition to or as an alternative to the above effect.

A preferable embodiment of the present disclosure will be described hereinafter in detail with reference to the accompanying drawings. It is noted that, in the present specification and drawings, repetitive description will be omitted by denoting constituent elements having substantially identical functional configurations by the same reference sign.

Furthermore, in the present specification and drawings, a plurality of constituent elements having substantially the same or similar functional configurations is sometimes distinguished from one another by putting different numbers after the same reference sign. However, in the case of no need to particularly distinguish the plurality of constituent elements having substantially the same or similar functional configurations, the constituent elements are denoted only by the same reference sign. Moreover, similar constituent elements in different embodiments are sometimes distinguished from one another by putting different alphabets after the same reference sign. However, in the case of no need to particularly distinguish the similar constituent elements, the constituent elements are denoted only by the same reference sign.

It is noted that the description is given in the following order.

Before starting description of an embodiment according to the present disclosure, circumstances until the inventors of the present disclosure reached to creation of the embodiment according to the present disclosure will first be described.

As previously described, in recent years, with the development of miniaturization, simplification, and the like of various types of motion sensors and biological information sensors, it has become easier to acquire various types of sensing data with these sensors, and the sensing data have been used, for example, for assistance of user's learning of a performance (playing of a musical instrument, a sport, and the like). In such learning assistance, data that can be used in the learning assistance is obtained by acquiring various pieces of sensing data from a plurality of sensors simultaneously in real time and analyzing the plurality of acquired sensing data.

Furthermore, in the case of performing such analysis, it is required to match timings of acquiring the sensing data and sensing time widths of the plurality of sensors with one another, that is, to synchronize the plurality of pieces of sensing data with one another. It is assumed that "synchronize/synchronization" means to match the timings of acquiring sensing data and the sensing time widths of the plurality of sensing data with one another in the following description.

Meanwhile, the sensing data from the biological information sensors are normally in an analog form such as waveform data varying with the passage of time. On the other hand, many sensors or electronic musical instruments and the like used by a user normally output sensing data or data in a digital form. To cope with the difference, in the learning assistance as described above, for purposes of simultaneously analyzing these plurality of pieces of sensing data, the sensing data in the analog form is converted into sensing data in the digital form by an analog-digital signal converter or the like before analysis. It is, therefore, required to accurately synchronize even the plurality of pieces of sensing data in a mixture of forms as described above so as to acquire data useful for the assistance of user's learning of the performance.

However, the plurality of pieces of sensing data is subjected to conversion as described above due to the difference in form; thus, it is difficult to accurately synchronize these plurality of pieces of sensing data.

In light of such circumstances, therefore, the inventors of the present disclosure have reached to creation of the embodiment of the present disclosure capable of accurately synchronizing and handling sensing data in a mixture of forms and available for the assistance of learning of a performance. An information processing system and an information processing method according to the embodiment of the present disclosure described above will be described hereinafter in sequence and in detail.

It is noted that the embodiment of the present disclosure will be described hereinafter regarding the case of applying the embodiment to guidance (learning of a skill) of a piano playing technique. However, application of the embodiment of the present disclosure is not limited to the guidance of the piano playing technique, and the embodiment of the present disclosure is applicable to playing techniques of other musical instruments, learning of skills of sports and traditional arts and crafts, rehabilitation of an impaired motility function, and the like. Furthermore, in the embodiment of the present disclosure, the musical instrument is not limited to the piano but may be every type of electronic musical instrument or every type of acoustic musical instrument.

Furthermore, in the following description, examples of a user include a learner receiving guidance of the piano playing technique and a performer who gives or is to give a performance such as a performer (for example, professional pianist) regarding whom various types of sensing data are collected in constructing a database according to the present embodiment, and also include an instructor or the like using the information processing system according to the present embodiment.

Moreover, in the following description, it is assumed that evaluation values (evaluation information) for a piano performance (performance) mean a performance speed (a pace, a rhythm) of the performer in the performance, accuracy of the performance (accuracy of the rhythm or of a volume of each tone), a volume of a sound generated by the performance (a peak of a sound waveform), a sound vibration (an integral value of the sound waveform), a timbre (a spectrum), a volume difference and a temporal difference (so-called "balance" in a chord (a group of musical notes)) among musical tones in the chord, a difference (range) between a maximum value and a minimum value of each sound parameter, a granularity (a resolution) of each sound parameter, energy efficiency in the performance, and the like. Furthermore, in the following description, parameters in a trade-off relation (for example, the performance speed and the accuracy) are present among the plurality of parameters described above; thus, in consideration of such a case, the evaluation values of the piano performance may be proportions of the parameters (numerical values). In general, an instructor gives guidance while placing greater importance on preferentially learning the performance accuracy than on the performance speed for the following reasons. In a case in which the learner repeats performances with many false motions and low accuracy while keeping a high performance speed, the body, cranial nerves, and the like of the learner are caused to learn the false motions. On the other hand, in the case of guidance with excessive importance placed on learning the accuracy, the learner strains himself/herself and falls into a habit of hardening the muscles, causing a reduction in energy efficiency in the performance. In the present embodiment, therefore, the proportions of the parameters may be used as the evaluation values in consideration of the balance of the parameters in the trade-off relation as described above. Moreover, in the case of applying the embodiment of the present disclosure not only to the piano performance but also to other user motions, the evaluation values of the performance may be a motion pattern of a motion element made by the user, a motion velocity, motion accuracy, a quantity of motion (motion power, an impulse, an amount of work, and the like), energy efficiency in the performance, a state of a result generated by the performance, and the like.

Furthermore, in the following description, synchronizing a plurality of pieces of sensing data means to match timings of acquisition and processing of the plurality of pieces of sensing data with one another and to match time widths of the plurality of pieces of sensing data with one another.

Additionally, in the following description, performing one test piece (for example, a music, a phrase, a scale, an arpeggio, a chord, or the like) by one performer or one learner is referred to as "one trial. " It is noted that, in the embodiment of the present disclosure, it is possible to acquire a plurality of pieces of sensing data in one trial.

A schematic configuration of an information processing system (information processing apparatus) <NUM> according to the embodiment of the present disclosure will first be described with reference to <FIG> is an explanatory diagram of an example of a configuration of the information processing system <NUM> according to the present embodiment. In the information processing system <NUM> according to the present embodiment, a server (information processing section) <NUM> analyzes a plurality of pieces of sensing data obtained from a plurality of sensor apparatuses (sensors) <NUM> that senses a state related to a performance given by moving of a user (learner or performer). Furthermore, the information processing system <NUM> feeds back information in various forms to the user via a feedback apparatus (information output section) <NUM> on the basis of a result of the analysis described above.

As depicted in <FIG>, the information processing system <NUM> according to the present embodiment includes the sensor apparatus <NUM>, the server <NUM>, and the feedback apparatus <NUM>, and these apparatuses are communicably connected to one another via a network <NUM>. For example, the sensor apparatus <NUM>, the server <NUM>, and the feedback apparatus <NUM> may be connected to the network <NUM> via base stations or the like not depicted (for example, base stations of cellular telephones or access points of a wireless LAN (Local Area Network)). In other words, any scheme, regardless of wired or wireless, can be applied to a communication scheme used in the network <NUM>. An outline of each apparatus included in the information processing system <NUM> according to the present embodiment will be described hereinafter.

The sensor apparatus <NUM> can sense a state related to a performance according to a motion of the performer or the learner. More specifically, the sensor apparatus <NUM> can be every type of biological information sensor that can be attached to a part of a body of the learner or the performer, an imaging apparatus that images the learner or the performer, a pressure sensor or a photoreflector sensor provided in the piano played by the learner or the performer, a sound pick-up apparatus (for example, a microphone) that collects the sound of the piano, or the like. Furthermore, the sensor apparatus <NUM> may be an electronic musical instrument such as an electronic piano per se capable of outputting signals. Moreover, the number of the sensor apparatuses <NUM> and a type thereof included in the information processing system <NUM> according to the present embodiment are not limited to a specific number and a specific type. It is noted that details of the sensor apparatus <NUM> will be described later.

The server <NUM> is configured with, for example, a computer. More specifically, the server <NUM> collects the sensing data related to the performance of the performer or the learner from the sensor apparatus <NUM>, analyzes and processes the collected sensing data, and outputs information for feedback (feedback information) to the learner or the like on the basis of results of analysis and processing. Furthermore, the server <NUM> may be, for example, a computer owned by the user, or a computer owned by a service provider that provides services by the present embodiment and that is present in a location different from a location of the user. It is noted that details of the server <NUM> will be described later.

The feedback apparatus <NUM> is an apparatus for presenting (outputting) the feedback information from the server <NUM> to the learner or the like, and notifies the learner of the feedback information (outputs the feedback information to the learner) as visible, haptic, auditory, or audible data during or after the performance of the learner. For example, the feedback apparatus <NUM> can be a device including a display device (display) and an audio output device (speaker) such as a tablet, a smart phone, a laptop PC (Personal Computer), or a notebook PC. Furthermore, the feedback apparatus <NUM> may be a wearable device that can be attached to a part of the body of the learner. More specifically, as the wearable device, every type of wearable device such as an HMD (Head Mounted Display) type, an ear device (headphone) type, an anklet type, a wristband type, a choker ring type, an eyewear type, a glove type, a pad type, a badge type, and a garment type. It is noted that details of the feedback apparatus <NUM> will be described later.

While <FIG> depicts that the information processing system <NUM> according to the present embodiment includes one sensor apparatus <NUM> and one feedback apparatus <NUM>, the configuration of the information processing system <NUM> according to the present embodiment is not limited to that depicted in <FIG>. In the present embodiment, the information processing system <NUM> can include, for example, a plurality of sensor apparatuses <NUM> and a plurality of feedback apparatuses <NUM>. Furthermore, the information processing system <NUM> according to the embodiment may include, for example, another communication apparatus such as a relay apparatus used at the time of transmitting the sensing data from the sensor apparatus <NUM> to the server <NUM>.

A configuration of the sensor apparatus <NUM> according to the embodiment of the present disclosure will next be described with reference to <FIG> is a block diagram depicting the configuration of the sensor apparatus <NUM> according to the present embodiment. As depicted in <FIG>, the sensor apparatus <NUM> according to the present embodiment mainly has a sensor section <NUM>, a main control section <NUM>, and a communication section <NUM>. Details of the functional sections of the sensor apparatus <NUM> will be described hereinafter.

The sensor section <NUM> can acquire, when attached to the body of the performer or the learner, the sensing data indicating a state of each motion element made by each part of the body of the learner or the performer during the performance. For example, the sensor section <NUM> is realized by one or a plurality of sensor devices including an acceleration sensor, an angular velocity sensor, a gyrosensor, a geomagnetic sensor, a position sensor, a vibration sensor, a pressure sensor, and a bending sensor. The sensor devices described above each detect changes in an acceleration, an angular velocity, or the like applied by the motion elements, and generate a plurality of pieces of sensing data indicating the detected changes. In a case in which the sensor section <NUM> is a sensor (first sensor) that acquires sensing data in an analog form, the sensing data is converted into sensing data in a digital form by a conversion apparatus (conversion section) <NUM> to be described later, and the sensing data in the digital form is output to the server <NUM>. However, in the present embodiment, the sensor section <NUM> described above is not limited to a sensor that acquires sensing data in the analog form but may be a sensor (second sensor) that acquires sensing data in the digital form.

Moreover, the sensor section <NUM> may be, for example, a key touch detection sensor that detects vertical movements of keys of the piano (a subject) moving by a motion (performance) of the learner or the performer. It is possible to detect the vertical movement of each key by installing the key touch detection sensor, for example, below each key. Specifically, the sensor section <NUM> can be, for example, a pressure sensor that detects a pressure applied to each key of the piano by the motion element of the learner or the performer, or a photoreflector sensor including a light receiving/emitting sensor that detects the vertical movement of each key by reflection of light. It is noted that, in a case in which the sensor section <NUM> as described above is the sensor (first sensor) that acquires the sensing data in the analog form, the sensing data is converted into the sensing data in the digital form by the conversion apparatus <NUM>, and the sensing data in the digital form is output to the server <NUM>. Moreover, in the present embodiment, the subject for which detection is performed is not limited to the keys of the piano but may be another musical instrument (acoustic musical instrument or electronic musical instrument) per se or a part of the other musical instrument.

Furthermore, in the present embodiment, the sensor section <NUM> may be an imaging apparatus that images the learner or the performer, and in this case, it is possible to quantitatively detect positions and motions of joints of the performer or the like by causing a high speed imaging camera (imaging apparatus) to capture motions of the performer or the like. Moreover, in the present embodiment, the imaging apparatus may detect a motion of an eyeball (eyeball movement) or a size of a pupil (pupil diameter) of the learner or the performer. It is noted that, in the case in which the sensor section <NUM> as described above is the sensor (first sensor) that acquires the sensing data in the analog form, the sensing data is converted into the sensing data in the digital form by the conversion apparatus <NUM>, and the sensing data in the digital form is output to the server <NUM>, similarly to the description given so far.

Moreover, in the present embodiment, the sensor section <NUM> may be a nuclear magnetic resonance sensor that detects an oral cavity state or an intratracheal state, a motion of a lip or a tongue, and the like of the learner or the performer using nuclear magnetic resonance. Specifically, the sensor section <NUM> can detect the state, the motion, and the like described above by causing the learner or the like to execute a performance within an MRI (Magnetic Resonance Imaging) apparatus. Particularly in the case of applying the embodiment of the present disclosure to a playing technique for every type of brass instrument (a flute, an oboe, a clarinet, a trumpet, or the like), the MRI is useful since it is possible to detect the motion of the lip or the tongue that is difficult to detect by other methods. It is noted that, in the case in which the sensor section <NUM> as described above is the sensor (first sensor) that acquires the sensing data in the analog form, the sensing data is converted into the sensing data in the digital form by the conversion apparatus <NUM>, and the sensing data in the digital form is output to the server <NUM>, similarly to the description given so far.

Furthermore, in the present embodiment, the sensor section <NUM> may be a biological information sensor such as a myoelectric sensor, a heartbeat sensor, a pulse sensor, a blood flow sensor, a respiration sensor, a brain wave sensor, a skin temperature sensor, a skin electrical conductivity (skin resistance) sensor, or a perspiration sensor. It is noted herein that the myoelectric sensor is a sensor that senses a feeble electrical field generated from muscle fibers configuring muscles. More specifically, the myoelectric sensor can quantitatively detect muscle active masses of the muscles by measuring myoelectric potentials by electrical signals that are generated in muscle fibers when the muscles of an arm and the like of the performer or the learner contract and that are propagated through a surface of the body by a plurality of electrodes attached to the arm and the like. Furthermore, the heartbeat sensor is a sensor that detects a heartbeat that is a beat of a heart, and the pulse sensor is a sensor that detects a pulse that is an arterial pulsation appearing on the surface of the body or the like when pressure changes occur in artery linings by feeding a blood to an entire body through arteries. The blood flow sensor is a sensor that emits an infrared radiation to the body and that detects a blood flow rate by reflection of the infrared radiation. The respiration sensor can be a respiratory flowmeter that detects a change in a respiratory volume. The brain wave sensor is a sensor that detects a brain wave by attaching a plurality of electrodes to a scalp, removing noise from a fluctuation in a measured potential difference between the electrodes, and extracting a periodic wave. The skin temperature sensor is a sensor that detects a body temperature of the performer or the learner, and the skin electrical conductivity sensor is a sensor that detects a skin electrical resistance of the performer or the learner. Moreover, the perspiration sensor is a sensor that detects perspiration of the performer or the learner. It is noted that, in the case in which the sensor section <NUM> as described above is the sensor (first sensor) that acquires the sensing data in the analog form, the sensing data is converted into the sensing data in the digital form by the conversion apparatus <NUM>, and the sensing data in the digital form is output to the server <NUM>, similarly to the description given so far.

Furthermore, the sensor section <NUM> may be a sound pick-up apparatus that collects a sound from the piano played by the performer or the learner. The sensor section <NUM> may be, for example, a microphone provided in the vicinity of the piano. It is noted that, in the case in which the sensor section <NUM> as described above is the sensor (first sensor) that acquires the sensing data in the analog form, the sensing data is converted into the sensing data in the digital form by the conversion apparatus <NUM>, and the sensing data in the digital form is output to the server <NUM>, similarly to the description given so far.

Moreover, the sensing data from the sensor section <NUM> may be an output (sound data) from an electronic musical instrument played by the learner or the performer, that is, used in the performance. In other words, in the present embodiment, the sensor section <NUM> may be an electronic musical instrument. In this case, the sensing data from the sensor section <NUM> is data in the digital form compliant with, for example, a MIDI (Musical Instrument Digital Interface) standard.

Furthermore, the sensor section <NUM> may include a position information sensor such as a GPS (Global Positioning System) receiver or the like that acquires position information regarding the learner or the performer. In addition, the sensor section <NUM> may include other various types of sensors such as an atmospheric pressure sensor, a temperature sensor, and a humidity sensor for acquiring environmental information indicating a state of an environment where the learner or the performer gives the performance.

Moreover, in the present embodiment, the sensor section <NUM> may be every type of sensor already provided in a musical instrument (acoustic musical instrument or an electronic musical instrument) already shipped by each musical instrument manufacturer.

The main control section <NUM> is provided in the sensor apparatus <NUM> and can exercise control over the blocks in the sensor apparatus <NUM>. The main control section <NUM> is realized by hardware which is, for example, a CPU (Central Processing Unit), a ROM (Read Only Memory), and a RAM (Random Access Memory). Furthermore, the main control section <NUM> can function as a data acquisition section <NUM>, a processing section <NUM>, and an output control section <NUM>. Details of these functional sections of the main control section <NUM> according to the present embodiment will be described hereinafter.

The data acquisition section <NUM> exercises control over the sensor section <NUM>, acquires the sensing data output from the sensor section <NUM>, and outputs the acquired sensing data to the processing section <NUM> to be described later.

The processing section <NUM> converts the sensing data output from the data acquisition section <NUM> described above into sensing data in a predetermined form in which the sensing data can be transmitted via the network <NUM>, and outputs the sensing data in the predetermined form to the output control section <NUM> to be described later.

The output control section <NUM> exercises control over the communication section <NUM> to be described later in such a manner as to transmit the sensing data in the predetermined form output from the processing section <NUM> described above to the server <NUM>.

The communication section <NUM> is provided in the sensor apparatus <NUM>, and can transmit and receive information to and from an external apparatus such as the server <NUM>. In other words, the communication section <NUM> can be said as a communication interface having functions to transmit and receive data. It is noted that the communication section <NUM> is realized by communication devices such as a communication antenna, a transmitting-receiving circuit, and a port.

It is noted that the sensor apparatus <NUM> may be a wearable device of every type including the HMD type, the ear device type, the anklet type, the wristband type, the choker ring type, the eyewear type, the pad type, the badge type, a belt type, and the garment type. Specifically, these wearable devices can be provided, as motion capture devices, on a finger, an arm, a leg, the body, a head, and a toe of the learner or the performer in order to acquire various kinds of sensing data. Furthermore, the sensor apparatus <NUM> may be an apparatus, such as the imaging apparatus or the sound pick-up apparatus, installed around the learner or the performer, or may be the musical instrument per se used by the learner or the performer; thus, the sensor apparatus <NUM> is not limited to a specific type. Moreover, in the present embodiment, the sensor apparatus <NUM> is not limited to the configuration depicted in <FIG>.

A configuration of the server <NUM> according to the embodiment of the present disclosure will next be described with reference to <FIG> is a block diagram depicting the configuration of the server <NUM> according to the embodiment of the present disclosure. As previously described, the server <NUM> is configured with, for example, a computer. As depicted in <FIG>, the server <NUM> mainly has an input section <NUM>, an output section <NUM>, a main control section <NUM>, a communication section <NUM>, and a storage section <NUM>. Details of the functional sections of the server <NUM> will be described hereinafter.

The input section <NUM> receives inputs of data and commands transmitted to the server <NUM>. More specifically, the input section <NUM> is realized by a touch panel, a keyboard, and the like, and can receive inputs of attribute information regarding the learner or the performer, subjective evaluations for the performance to be described later, and the like.

The output section <NUM> is configured with, for example, a display, a speaker, a video output terminal, an audio output terminal, and the like, and outputs various types of information by images or audio.

The main control section <NUM> is provided in the server <NUM> and can exercise control over the blocks in the server <NUM>. The main control section <NUM> is realized by hardware which is, for example, a CPU, a ROM, and a RAM. Furthermore, the main control section <NUM> can function as a data acquisition section <NUM>, an analysis section (information processing section) <NUM>, and an output control section (information output section) <NUM>. Details of these functional sections of the main control section <NUM> according to the present embodiment will be described hereinafter.

The data acquisition section <NUM> acquires the sensing data transmitted from the sensor apparatus <NUM> described above, and outputs the acquired sensing data to the processing section <NUM> to be described later.

The processing section <NUM> analyzes and processes the sensing data output from the data acquisition section <NUM> described above. The output control section <NUM> then generates feedback information to be fed back to the user on the basis of results of analysis and processing. Furthermore, the processing section <NUM> can construct a database (DB) <NUM> (refer to <FIG>) including various types of information for generating the feedback information. More specifically, to realize these functions described above, the processing section <NUM> functions as an analysis section <NUM>, a learning section <NUM>, and a comparison section <NUM> as depicted in <FIG>. Details of these functional sections of the processing section <NUM> according to the present embodiment will be described hereinafter.

The analysis section <NUM> performs analysis and the like on a plurality of pieces of sensing data that can be acquired in one trial, and extracts feature variables characterizing a state of the performance of the learner or the performer. The feature variables can be extracted as a maximum value, a minimum value, a mean value, an integral value, a period, an amount of change of each sensing data by statistically processing each sensing data by the analysis section <NUM>. More specifically, the feature variables can be timings of peaks of muscle activities and maximum joint angles, and the like. The feature variables extracted in this way are linked to the plurality of pieces of sensing data acquired in the trial and stored in the storage section <NUM> to be described later as the DB <NUM>. Furthermore, the analysis section <NUM> may output the extracted feature variables to the learning section <NUM>, the comparison section <NUM>, and the like to be described later.

Moreover, the analysis section <NUM> may perform analysis and the like on the plurality of pieces of sensing data (more specifically, for example, audio data related to the performance) that can be acquired in one trial, and extract evaluation values for the performance of the learner or the performer. The evaluation values are, for example, evaluation values for a piano performance, and can be the performance speed of the performer in the performance, the accuracy of the performance, the volume of the sound generated by the performance, the sound vibration, the timbre, the volume difference and the temporal difference among the musical tones in the chord, the difference between the maximum value and the minimum value of each sound parameter, the granularity of each sound parameter, and the like. The evaluation values extracted in this way are linked to the plurality of pieces of sensing data acquired in the trial and stored in the storage section <NUM> to be described later as the DB <NUM>. Furthermore, the analysis section <NUM> may output the extracted evaluation values to the learning section <NUM>, the comparison section <NUM>, and the like to be described later.

The learning section <NUM> acquires information associated with relation among the sensing data, the feature variables, and the evaluation values by performing multivariate analysis thereon. More specifically, the learning section <NUM> is a supervised learning instrument such as a support vector regression and a deep neural network, and can perform machine learning on the relation between the sensing data and the evaluation values by performing, for example, the multivariate analysis such as multiple regression analysis thereon. Information regarding the relation (relation information) obtained by performing the machine learning by the learning section <NUM> can be stored in the storage section <NUM> to be described later as the DB <NUM>, and can be used in selection of training data performed by the comparison section <NUM> to be described later.

The comparison section <NUM> selects one or a plurality of pieces of sensing data that serves as a model of the learner from the DB <NUM>, and compares the selected sensing data (training data) with sensing data (comparison data) of the same item out of a plurality of pieces of newly acquired sensing data regarding the learner. More specifically, the comparison section <NUM> may calculate, for example, a difference (gap) between the training data and the comparison data or calculate a degree of matching of these pieces of data. Furthermore, the comparison section <NUM> may perform comparison by superimposing the training data on the comparison data.

Specifically, the comparison section <NUM> selects, as the training data, for example, representative sensing data that serves as the model of the learner aiming at an accurate performance from the DB <NUM>, and compares the selected training data with the sensing data regarding the learner of the same item. At this time, in the case of presence of a plurality of pieces of representative sensing data serving as models, the comparison section <NUM> calculates a difference between each sensing data and the sensing data regarding the learner of the same item as an item of the representative sensing data. The comparison section <NUM> then selects the sensing data greatest in difference as the sensing data related to an element with high necessity of correction such that the performance of the learner becomes more accurate. In the present embodiment, such selection makes it possible to pay more attention on a technical element which the learner is slowest in learning; thus, it is possible to generate the feedback information useful to efficiently acquire the accurate performance.

Alternatively, the comparison section <NUM> may select, as the training data, the sensing data estimated to have a higher relation with the accuracy (evaluation value) of the performance on the basis of the information regarding the relation obtained by the learning section <NUM> described above. In the present embodiment, such selection makes it possible to pay more attention on a technical element that enables efficient acquisition of the accurate performance; thus, it is possible to generate the feedback information useful to efficiently acquire the accurate performance.

In another alternative, in the present embodiment, the comparison section <NUM> may select, as the training data, the sensing data related to a performer having the same or similar attribute information as or to attribute information (a gender, an age, a physical size, a muscle force, tenderness, legerity, and the like) regarding the learner on the basis of the attribute information regarding the learner. In the present embodiment, such selection makes it possible to perform feedback tailormade to attributes and the like of the learner; thus, the learner can efficiently perform learning. Moreover, in the case of comparison between a past state and a current state of the performance of the learner, the comparison section <NUM> may select past sensing data regarding the learner.

Moreover, in the present embodiment, the comparison section <NUM> is not limited to comparing the sensing data but may compare the feature variables extracted by the analysis section <NUM> described above. Even in such a case, the comparison section <NUM> can perform the selection described above in a case in which a plurality of representative feature variables serving as models is present.

The output control section <NUM> generates the feedback information on the basis of a result of comparison by the comparison section <NUM>, and exercises control over the communication section <NUM> to be described later in such a manner as to transmit the feedback information to the feedback apparatus <NUM>. The transmitted feedback information is output to the user via the feedback apparatus <NUM>. Furthermore, the output control section <NUM> may perform processing for emphasizing or reducing the difference (gap) between the training data and the comparison data by expanding or contracting the difference either spatially or temporally, and generate the feedback information. Specifically, in the present embodiment, the output control section <NUM> feeds back the feedback information generated by reducing the difference to the learner at a technical level greater in difference from the training data to avoid decrease of motivation of learning. On the other hand, in the present embodiment, the output control section <NUM> feeds back the feedback information generated by emphasizing the small difference to guide the learner to a higher technical level in such a manner that the learner can easily recognize the difference, to the learner at the technical level smaller in difference from that of the training data. In other words, in the present embodiment, performing the processing as described above makes it possible for the learner to easily recognize the difference from the training data while avoiding the decrease of the motivation of the learner; thus, it is possible to realize assistance of efficient learning of the performance.

Moreover, the output control section <NUM> may select a sensation modality (such as a visual sensation, an auditory sensation, or a tactile sensation) suited for the feedback information on the basis of user's situations and the like, and transmit the feedback information to the feedback apparatus <NUM> according to the selected sensation modality. In the present embodiment, the feedback information can be provided to the user by the sensation modality according to the user's situations and the like; thus, it is possible to realize assistance of efficient learning of the performance.

The communication section <NUM> is provided in the server <NUM>, and can transmit and receive information to and from an external apparatus such as the sensor apparatus <NUM> or the feedback apparatus <NUM>. It is noted that the communication section <NUM> is realized by communication devices such as a communication antenna, a transmitting-receiving circuit, and a port.

The storage section <NUM> is provided in the server <NUM> and stores therein programs, information, and the like for the main control section <NUM> described above to execute various types of processing. Furthermore, the storage section <NUM> stores therein the DB <NUM> including the plurality of pieces of sensing data linked to various types of attribute information and the like (a metafile). It is noted that the storage section <NUM> is realized by a magnetic recording medium such as a hard disk (HD), a nonvolatile memory such as a flash memory, or the like.

It is noted that the metafile (attribute information) can contain attribute information associated with the learner or the performer (the name, the gender, the age, a body height, a body weight, the physical size, the muscle force, a palm size, the tenderness, the legerity, years of experience of piano playing, a skill level, a national origin, an instructor name, and the like), attribute information associated with the performance (date and time of performance, a title of a music, a category of the music, a name of a composer, an epoch of composition, a category of the composer, a tempo, a volume, information whether a single tone or complexed tones, contents of teaching given to the performer or the learner, the musical instrument for the performance, a location of the performance, and the like), feature variables characterizing the performance, evaluation values (evaluation information) for the performance, attribute information associated with sensing (a sensor type, a sampling frequency, the number of channels, and the like), sensitivity evaluation information regarding the performance (information such as an image, a timbre, and a vibration the performer is to express), and the like. The metafile can also contain attribute information regarding each trial (a competition name, a competition level, an attendance, the number of fans, evaluation information regarding attendance, the number of views, and the like). In other words, the metafile functions as a label of the plurality of pieces of sensing data (sensing data group) acquired in one trial, and can indicate, for example, that the sensing data group is information sensed when a certain performer performs a certain piece while aiming to express the piece in a certain manner.

Furthermore, in the present embodiment, the server <NUM> is not limited to the configuration depicted in <FIG> but may include other functional blocks and the like.

Meanwhile, as previously described, the sensing data obtained by the sensor apparatus <NUM> included in the information processing system <NUM> according to the present embodiment is a mixture of the sensing data in the analog form and the sensing data in the digital form. To acquire data useful for assistance of learning of the performance of the learner, it is required to accurately synchronize a plurality of pieces of sensing data even in a mixture of forms described above.

In the present embodiment, therefore, it is preferable to provide the conversion apparatus (conversion section) <NUM> and a computing apparatus <NUM> between the sensor apparatus <NUM> and the server <NUM> as depicted in <FIG>. Therefore, the conversion apparatus <NUM> and the computing apparatus <NUM> according to the present embodiment will be described hereinafter with reference to <FIG> is a block diagram depicting configurations of the conversion apparatus <NUM> and the computing apparatus <NUM> according to the present embodiment. More specifically, the information processing system <NUM> according to the present embodiment includes the conversion apparatus <NUM> and the computing apparatus <NUM> as depicted in <FIG> between the sensor apparatus <NUM> and the server <NUM>. Details of the conversion apparatus <NUM> and the computing apparatus <NUM> will be described hereinafter.

As depicted in <FIG>, the conversion apparatus <NUM> is an apparatus that converts a plurality of pieces of sensing data in different forms obtained from a plurality of sensor apparatuses <NUM>, and mainly has a digital-analog signal conversion section <NUM> and an analog-digital signal conversion section <NUM>. Details of the digital-analog signal conversion section <NUM> and the analog-digital signal conversion section <NUM> of the conversion apparatus <NUM> will be described hereinafter.

The digital-analog signal conversion section <NUM> converts sensing data received from a sensor apparatus (second sensor) 10b that outputs sensing data in the digital form among the plurality of sensor apparatuses <NUM> via the computing apparatus <NUM>, to be described later, into sensing data in the analog form. Furthermore, the digital-analog signal conversion section <NUM> outputs the sensing data in the analog form obtained by conversion to the analog-digital signal conversion section <NUM> to be described later.

More specifically, the digital-analog signal conversion section <NUM> has a plurality of digital-analog converters (D/A converters) <NUM> including a microcomputer board and the like. For example, one sensing data acquired by one sensor apparatus 10b is extracted as sensing data in one of a plurality of channels by the computing apparatus <NUM> to be described later. The extracted sensing data per channel is input to each of the D/A converters <NUM>. Specifically, in a case in which the sensor apparatus 10b is an electronic piano that outputs a MIDI signal that is sensing data in the digital form compliant with the MIDI standard, each packet in the MIDI signal contains information regarding an event such as key touch/key release (a status), information regarding a pitch of a generated sound (a note), and information regarding a loudness of the sound (a velocity). In the present embodiment, therefore, the computing apparatus <NUM> extracts the MIDI signal as pieces of sensing data regarding the status, the note, and the velocity as described above per channel, and inputs the extracted sensing data to the D/A converters <NUM>. Moreover, each extracted sensing data is converted into sensing data in the analog form by each D/A converter <NUM>, and the sensing data in the analog form is input to the analog-digital signal conversion section <NUM> to be described later. In the embodiment, converting the sensing data in the digital form into the sensing data in the analog form in this way makes it possible to simultaneously handle the sensing data in the analog form with the sensing data that is in the analog form from the first start and that is output from a sensor apparatus 10a.

Moreover, a part of the sensing data the form of which is converted into the analog form by the digital-analog signal conversion section <NUM> contains, for example, information regarding a timing of occurrence of an event such as a moment of touching specific keys at a strength equal to or higher than a certain level (event). Therefore, sensing data containing such an event may be converted into sensing data in the analog form by the digital-analog signal conversion section <NUM>, and input to the other sensor apparatus <NUM> (for example, the sensor apparatus (first sensor) 10a that acquires the sensing data in the analog form) as a trigger signal. The trigger signal can be used for the sensor apparatus 10a to start (drive) acquiring the sensing data.

The analog-digital signal conversion section <NUM> converts sensing data from the sensor apparatus (first sensor) 10a that outputs sensing data in the analog form among the plurality of sensor apparatuses <NUM> into sensing data in the digital form. Furthermore, the analog-digital signal conversion section <NUM> converts the sensing data in the analog form into which the digital-analog signal conversion section <NUM> described above converts the sensing data in the digital form, into sensing data in the digital form. The analog-digital signal conversion section <NUM> then outputs the sensing data obtained by conversion to the server <NUM> described above. More specifically, the analog-digital signal conversion section <NUM> has a plurality of analog-digital converters (A/D converters) <NUM> including a microcomputer board and the like, and each A/D converter <NUM> can convert each sensing data in the analog form into the sensing data in the digital form and output the sensing data in the digital form to the server <NUM>.

Furthermore, the analog-digital signal conversion section <NUM> has a clock (first clock mechanism) <NUM> mounted on the microcomputer board to synchronize a plurality of pieces of sensing data. The clock <NUM> simultaneously outputs a trigger signal to each of the A/D converters <NUM> or each sensor apparatus 10a, and causes the A/D converter <NUM> or the sensor apparatus 10a to start acquiring the sensing data by the trigger signal, thereby making it possible to match timings of acquiring the sensing data. In other words, in the present embodiment, the clock <NUM> enables the pieces of sensing data from the plurality of sensing apparatuses 10a to be accurately synchronized with one another. It is noted that the clock <NUM> may output clock time information to the server <NUM>.

The computing apparatus <NUM> acquires the sensing data from the sensor apparatus 10b that acquires the sensing data in the digital form, and extracts the acquired sensing data as sensing data per channel (containing, for example, the status, the note, and the velocity). Furthermore, the computing apparatus <NUM> outputs the sensing data extracted for each of a plurality of channels to the conversion apparatus <NUM> described above. More specifically, the computing apparatus <NUM> mainly has a receiving section <NUM> and a computing section <NUM> as depicted in <FIG>. Details of the receiving section <NUM> and the computing section <NUM> of the computing apparatus <NUM> will be described hereinafter.

The receiving section <NUM> receives the sensing data from the sensor apparatus 10b that acquires the sensing data in the digital form, and outputs the sensing data in the digital form to the computing section <NUM> to be described later.

The computing section <NUM> extracts the sensing data acquired from the receiving section <NUM> described above as the sensing data per channel. For example, as described above, the computing section <NUM> extracts the status, the note, and the velocity contained in each packet in the MIDI signal that is the sensing data in the digital form as the sensing data per channel. Furthermore, the computing section <NUM> outputs the extracted sensing data per channel to the conversion apparatus <NUM> described above.

Furthermore, the computing section <NUM> may have a clock (second clock mechanism) <NUM> mounted on a microcomputer board to synchronize the plurality of pieces of sensing data. The clock <NUM> can link clock time information to the sensing data. In the present embodiment, since the clock <NUM> links the sensing data in the plurality of channels from the sensor apparatus 10b to the same clock time information, it is possible to match clock times of the pieces of sensing data with one another, that is, it is possible to synchronize the pieces of sensing data. It is noted that the clock <NUM> may output the clock time information to the server <NUM>.

Furthermore, the server <NUM> can synchronize the plurality of pieces of sensing data from the conversion apparatus <NUM> on the basis of the clock time information from the clocks <NUM> and <NUM>. More specifically, the sensing data from the sensor apparatus 10b that acquires the sensing data in the digital form is converted into the sensing data in the analog form in advance, and then the sensing data in the analog form is converted into the sensing data in the digital form. On the other hand, the sensing data from the sensor apparatus 10a that acquires the sensing data in the analog form is converted into the sensing data in the digital form without being converted into the sensing data in the analog form in advance. Therefore, a minor temporal difference is generated between the sensing data from the sensor apparatus 10b and the sensing data from the sensor apparatus 10a at a point in time of arrival of the sensing data at the server <NUM>. In the present embodiment, therefore, to accurately synchronize the sensing data from the sensor apparatus 10b with the sensing data from the sensor apparatus 10a, the server <NUM> performs temporal shift processing on these pieces of sensing data on the basis of a temporal difference between the clocks <NUM> and <NUM> obtained from the clock time information. According to the present embodiment, therefore, it is possible to match time bases for the plurality of pieces of sensing data in a mixture of forms and accurately synchronize and handle these pieces of sensing data.

It is noted that the clock <NUM> is not necessarily provided in the computing apparatus <NUM> and may be provided in the sensor apparatus 10b or in a portion of the digital-analog signal conversion section <NUM> closer to the computing apparatus <NUM>, that is, closer to the sensor apparatus 10b.

It is noted that, in the present embodiment, the configurations of the conversion apparatus <NUM> and the computing apparatus <NUM> are not limited to those depicted in <FIG>. For example, both or one of the conversion apparatus <NUM> or the computing apparatus <NUM> may be provided in the server <NUM> described above. Alternatively, the conversion apparatus <NUM> may be provided in the sensor apparatus 10a, or the computing apparatus <NUM> may be provided in the sensor apparatus 10b; thus, the configurations of the conversion apparatus <NUM> and the computing apparatus <NUM> are not limited to specific ones. Nevertheless, providing any one of the conversion apparatus <NUM> or the computing apparatus <NUM> in the sensor apparatus 10a or 10b means that a weight of the sensor apparatus <NUM> attached to the body of the performer or the learner or attached to the subject moving according to the motion of the performer or the learner becomes heavier. In addition, it is estimated that the weight has an influence on the performance of the performer or the learner. Moreover, in such a case, it is estimated that the sensing data from the sensor apparatuses 10a and 10b tend to contain noise. It is, therefore, preferable that the conversion apparatus <NUM> and the computing apparatus <NUM> are provided in locations other than the sensor apparatuses 10a and 10b to avoid the influence on the performance and the noise.

As described so far, by having the conversion apparatus <NUM> described above, the information processing system <NUM> according to the present embodiment can accurately synchronize the pieces of sensing data even in the case of a mixture of the sensing data in the analog form and the sensing data in the digital form. As a result, the information processing system <NUM> can simultaneously analyze the plurality of pieces of sensing data accurately synchronized with one another and containing various information; thus, it is possible to acquire data useful for assistance of learning of the performance of the learner.

A configuration of the feedback apparatus <NUM> according to the embodiment of the present disclosure will next be described with reference to <FIG> is a block diagram depicting the configuration of the feedback apparatus <NUM> according to the present embodiment. As previously described, the feedback apparatus <NUM> can be a device such as the tablet, the laptop PC, the notebook PC, or the wearable device. Furthermore, as depicted in <FIG>, the feedback apparatus <NUM> mainly has a haptic mechanism <NUM>, a display section <NUM>, an audio output section <NUM>, a main control section <NUM>, a communication section <NUM>, and a storage section <NUM>. Details of the functional sections of the feedback apparatus <NUM> will be described hereinafter.

The haptic mechanism <NUM> is an apparatus that propagates a sense of force (tactile sensation), for example, applies a force to a part (for example, a joint) of the body of the learner on the basis of the feedback information transmitted from the server <NUM>. For example, the haptic mechanism can be a glove type wearable device (wearable apparatus) attached to a hand of the learner. Furthermore, in the present embodiment, the haptic mechanism <NUM> is not limited to the glove type wearable device and may be, for example, a vibration apparatus that applies a vibration to a part of the body of the learner or a stimulation apparatus that uses electrical muscle stimulation to give a stimulus to muscles. In other words, in the present embodiment, it is sufficient if the haptic mechanism <NUM> is capable of sensorily feeding back the feedback information to the learner (performing bio-feedback) by giving a tactile stimulus to a part of the body of the learner.

The display section <NUM> is a device for displaying the feedback information to the user, and outputs the feedback information toward the user by, for example, images or light. The display section <NUM> is realized by, for example, a display, a light-emitting element (not depicted), and the like. Furthermore, the display section <NUM> may be realized by a video output terminal and the like.

Moreover, the display section <NUM> may be a projection apparatus that can display an object based on feedback information <NUM> by superimposing the object onto a real space as augmented reality (AR). Such a projection apparatus can be, for example, a smart glass type wearable device attached in front of learner's eyes. A transmissive display is provided in the smart glass type wearable device, and the transmissive display holds a virtual image optical system including a transparent light guide section and the like in front of the learner's eyes using, for example, a half-mirror and a transparent light guide plate, and displays the object inside of the virtual image optical system. Furthermore, the projection apparatus may be an HMD attached to the learner's head.

The audio output section <NUM> is a device that outputs the feedback information to the user as audio, and may be, for example, a headphone speaker attached to a learner's ear or a speaker (not depicted) provided in the vicinity of the learner. Furthermore, the audio output section <NUM> may be realized by an audio output terminal and the like.

In this way, in the present embodiment, means corresponding to a suited sensation modality is selected from among the haptic mechanism <NUM>, the display section <NUM>, and the audio output section <NUM> according to the feedback information and the like and the user's situations and the like, and the feedback information can be fed back to the learner or the like. Moreover, in the present embodiment, the haptic mechanism <NUM>, the display section <NUM>, and the audio output section <NUM> may simultaneously perform feedback by a plurality of sensation modalities, and a method of the feedback is not limited to a specific method.

The main control section <NUM> is provided in the feedback apparatus <NUM> and can exercise control over the blocks in the feedback apparatus <NUM>. The main control section <NUM> is realized by hardware which is, for example, a CPU, a ROM, and a RAM.

The communication section <NUM> can transmit and receive information to and from an external apparatus such as the server <NUM>. It is noted that the communication section <NUM> is realized by communication devices such as a communication antenna, a transmitting-receiving circuit, and a port.

The storage section <NUM> is provided in the feedback apparatus <NUM>, and stores therein programs and the like for the main control section <NUM> to execute various types of processing and information obtained by the processing. It is noted that the storage section <NUM> is realized by a magnetic recording medium such as an HD, a nonvolatile memory such as a flash memory, or the like.

Moreover, the feedback apparatus <NUM> may have an input section that is not depicted. The input section has a function to receive input of data and commands transmitted to the feedback apparatus <NUM>. More specifically, the input section is realized by a touch panel, buttons, switches, keys, a keyboard, a microphone, an image sensor, and the like.

Furthermore, in the present embodiment, a function of the sensor section <NUM> in the sensor apparatus <NUM> and the haptic mechanism <NUM> and the like in the feedback apparatus <NUM> may be integrated into one wearable device. It is noted that, in the present embodiment, the feedback apparatus <NUM> is not limited to the configuration depicted in <FIG> and may have, for example, other functional sections.

The configuration of the information processing system <NUM> according to the present embodiment and those of the apparatuses included in the information processing system <NUM> have been described so far in detail. An information processing method according to the present embodiment will next be described with reference to <FIG> is a flowchart illustrating an example of the information processing method according to the present embodiment.

As depicted in <FIG>, the information processing method according to the present embodiment includes a plurality of steps from Step S100 to Step S500. Details of the steps included in the information processing method according to the present embodiment will be described hereinafter.

First, the sensor apparatus <NUM> is attached to a part of the body of the learner or the performer in advance or installed around the learner or the like before the performance of the learner or the like. When the learner or the like then starts a predetermined performance (such as a piece of music, a phrase, a scale, an arpeggio, or a chord) as a trial, the sensor section <NUM> of the sensor apparatus <NUM> senses movements of the body, the musical instrument, and the like generated to accompany the performance (motion) of the learner or the like during the performance, a sound generated by the musical instrument, and the like, and acquires a plurality of pieces of sensing data. Furthermore, the sensor apparatus <NUM> outputs the acquired sensing data to the server <NUM>.

At this time, the sensing data acquired by the sensor apparatus <NUM> is converted by the conversion apparatus <NUM> described above and the converted sensing data is input to the server <NUM>. In the information processing system <NUM> according to the present embodiment, therefore, the conversion apparatus <NUM> can accurately synchronize the pieces of sensing data even in the case of the mixture of the sensing data in the analog form and the sensing data in the digital form, as previously described. As a result, the server <NUM> can simultaneously analyze the plurality of pieces of sensing data accurately synchronized and containing various types of information in a subsequently executed step; thus, it is possible to acquire data useful for assistance of the learning of the performance of the learner.

In the present embodiment, it is preferable to allow many performers (for example, approximately <NUM> performers) to each give a predetermined performance (such as a piece of music, a phrase, a scale, an arpeggio, or a chord) and to collect many pieces of sensing data to acquire information for constructing the DB <NUM>. Furthermore, at that time, the server <NUM> may acquire the attribute information associated with each performer (the gender, the age, the body height, the body weight, the physical size, the muscle force, the palm size, the tenderness, the legerity, the years of experience of piano playing, the skill level, and the like) and the attribute information associated with the performance (the date and time, the title of music, the name of a composer, the musical instrument for the performance, the location of the performance, and the like).

The server <NUM> generates training data on the basis of the sensing data regarding the performer acquired in Step S100. The generated training data is stored (archived) as the DB <NUM>. It is noted that details of Step S200 will be described later.

The server <NUM> generates comparison data on the basis of the sensing data regarding the learner acquired in Step S100. The generated comparison data may be stored as the DB <NUM>. It is noted that details of Step S300 will be described later.

The server <NUM> selects the training data from the DB <NUM> and compares the selected training data with the comparison data generated in Step S300 described above. The server <NUM> then generates feedback information to be fed back to the learner on the basis of a comparison result. It is noted that details of Step S400 will be described later.

The feedback apparatus <NUM> performs feedback to the learner on the basis of the feedback information generated in Step S400 described above.

The details of Steps S200, S300, and S400 depicted in <FIG> will be described hereinafter.

First, details of creation of the training data in Step S200 of <FIG> described above will be described with reference to <FIG>. <FIG> is a flowchart of creating the training data according to the present embodiment. <FIG> and <FIG> are explanatory diagrams of an example of extraction of feature variables <NUM>. Furthermore, <FIG> are explanatory diagrams of an example of construction of the database <NUM>, and <FIG> and <FIG> are explanatory diagrams of an example of creation of the training data.

As depicted in <FIG>, the creation of the training data includes a plurality of steps from Step S201 to Step S213. Details of these plurality of steps will be described hereinafter.

The server <NUM> acquires the plurality of pieces of sensing data (a sensing data group) acquired by each sensor apparatus <NUM> as one text file from start until end of sensing in one trial. A metafile is linked to the text file, and the metafile can contain the attribute information associated with the performer, the attribute information associated with the performance, and the like, as previously described. The metafile can also contain the attribute information regarding the trial (the competition name, the competition level, the attendance, the number of fans, the evaluation information regarding attendance, the number of views, and the like). In the present embodiment, linking the metafile to the sensing data group in this way makes it possible to appropriately and easily manage the sensing data group and select the training data by referring to various types of information contained in the metafile.

It is noted that the above information contained in the metafile may be input by the user to the server <NUM>, or may be generated by analyzing the sensing data and automatically extracting the information if the information can be analyzed from the sensing data. For example, a piece performed may be automatically extracted and the attendance present in the surroundings may be estimated by analyzing the sound information contained in the sensing data, and the location of the performance may be automatically extracted by analyzing the position information contained in the sensing data.

The server <NUM> removes noise and the like contained in each sensing data acquired in Step S201 by performing filtering and statistical processing (abnormal value detection) (preprocessing).

The server <NUM> performs analysis and the like on the plurality of pieces of sensing data acquired in one trial and preprocessed in Step S203 described above, and extracts feature variables characterizing the state of the performance. The feature variables extracted in this way are written to the metafile in Step S209 to be described later. Specifically, the analysis section <NUM> of the server <NUM> analyzes sensing data <NUM> and extracts feature variables <NUM>, as depicted in <FIG>. For example, the feature variables <NUM> can be extracted as the maximum value, the minimum value, the mean value, the integral value, the period, and the amount of change of each of the sensing data <NUM> by statistically processing the sensing data <NUM> by the analysis section <NUM>. More specifically, as the feature variables <NUM>, timings at which the keys are located at uppermost positions, timings at which a key touch ascent velocity becomes maximum, and timings at which a key touch descent velocity becomes maximum, as indicated by white circle, white triangle, and white rectangle markers, are extracted on the sensing data <NUM> that is time-series data regarding moving amounts of the keys depicted in <FIG>.

As depicted in <FIG>, the analysis section <NUM> of the server <NUM> may perform analysis and the like on the plurality of pieces of sensing data (for example, audio data related to the performance) <NUM> acquired in one trial and preprocessed in Step S203 as described above, and extract evaluation values 506a for the performance. The evaluation values 506a can be the performance speed of the performer in the performance, the accuracy of the performance, the volume of the sound generated by the performance, the sound vibration, the timbre, the volume difference and the temporal difference among the musical tones in the chord, the difference between the maximum value and the minimum value of each sound parameter, the granularity of each sound parameter, and the like, as previously described. The evaluation values 506a extracted in this way are written to the metafile in Step S209 to be described later, similarly to the feature variables <NUM>.

Furthermore, the server <NUM> acquires information regarding a sensitivity evaluation that is an evaluation of sensitivity for the performance that cannot be directly evaluated by the feature variables <NUM> and the evaluation values 506a described above. For example, the server <NUM> acquires the sensitivity evaluation by causing the performer to input the sensitivity for the own performance (brilliant, flowing, heavy, surrounding, lyrical, soft, hard, or the like) after end of the trial. The acquired sensitivity evaluation is written to the metafile in Step S209 to be described later, similarly to the feature variables <NUM>. It is noted that a method of acquiring the information regarding the sensitivity evaluation is not limited to the method by causing the performer to directly input the sensitivity, but the information regarding the sensitivity evaluation may be automatically acquired by audio analysis of murmurs or the like of the performer during the performance. In the present embodiment, automatically acquiring the information regarding the sensitivity evaluation by the audio analysis in this way makes it possible to acquire the information regarding the sensitivity evaluation without bothering the performer.

Moreover, in the present embodiment, the information regarding the sensitivity evaluation may be acquired by other methods. For example, many people (for example, professionals such as pianists and nonprofessionals other than the professionals) are asked to listen to various types of music (for example, performances of performers and performances of nonprofessionals, not limited to specific performances) in advance and to perform the sensitivity evaluation for each piece of music. Next, the server <NUM> inputs the sensitivity evaluation to the learning section <NUM> as a training signal, inputs the music to the learning section <NUM> as an input signal, and causes the learning section <NUM> to perform machine learning on a relation between the music and the sensitivity evaluation. By referring to the relation obtained by the machine learning by the learning section <NUM>, the server <NUM> can automatically acquire the information regarding the sensitivity evaluation for the performance in the trial on the basis of the music (vibration and the like) by the performance in the trial. It is noted that, in the present embodiment, weighting may be performed in such a manner that the sensitivity evaluation of the professionals is reflected in the relation obtained by the machine learning more strongly than that of the nonprofessionals, or weighting may be performed conversely in such a manner that the sensitivity evaluation of the nonprofessionals is reflected in the relation obtained by the machine learning more strongly than that of the professionals. Moreover, in the present embodiment, the server <NUM> may input parameters (a key touch force, a key touch timing, a key touch length, and the like) of the key touch on the piano by a robot or the like and a sensitivity evaluation by the robot or the like for the sound, the music, and the like by the key touch to the learning section <NUM>, cause the learning section <NUM> to perform machine learning on a relation between the sound and the like by the key touch by the robot or the like and the sensitivity evaluation by the robot or the like, and acquire the relation. In the present embodiment, referring to the machine learning in this way makes it possible to acquire the information regarding the sensitivity evaluation without bothering the performer.

The server <NUM> writes the feature variables <NUM>, the evaluation values 506a, and the information regarding the sensitivity evaluation acquired in Steps S205 and S207 to the metafile described above. In the present embodiment, linking the feature variables <NUM>, the evaluation values 506a, and the information regarding the sensitivity evaluation to the sensing data group makes it possible to appropriately and easily manage the sensing data group and select the training data.

As depicted in <FIG>, the server <NUM> repeats Steps S201 to S209 described above, collects a plurality of sensing data groups (text files) <NUM> and metafiles <NUM> linked to the respective sensing data groups <NUM>, and constructs the DB <NUM>. As previously described, each sensing data group <NUM> in the DB <NUM> can be managed on the basis of the information contained in the metafile <NUM>. As depicted in, for example, <FIG>, each sensing data group <NUM> can be managed per performer (such as a pianist A or a pianist B) in a performance chronological order on the basis of the information contained in the metafile <NUM>. It is noted that, in the present embodiment, a method of management is not limited to managing each sensing data group <NUM> per performer and that each sensing data group <NUM> can be managed under various conditions such as per title of music or per age.

The server <NUM> creates the training data on the basis of the database <NUM> constructed in Step S211 described above. In the present embodiment, the training data can be created mainly from two methods.

In a first method, the sensing data <NUM> or the feature variables <NUM> obtained in one trial are used as the training data as it is or as they are.

In a second method, machine learning is performed using a plurality of pieces of sensing data <NUM> regarding a plurality of performers or a plurality of trials, thereby using a result obtained by the machine learning as the training data.

As depicted in, for example, <FIG>, the server <NUM> inputs the sensing data groups <NUM> and the metafiles <NUM> each containing the evaluation values 506a to the learning section <NUM> as input signals and training signals, respectively. The learning section <NUM> performs machine learning on the relation between the plurality of pieces of sensing data <NUM> and the plurality of series of evaluation values 506a by performing multivariate analysis such as multiple regression analysis thereon. The learning section <NUM> may then output information regarding the relation (relation information) <NUM> obtained by performing the machine learning as the training data.

The plurality of pieces of sensing data <NUM> handled by the learning section <NUM> at this time are accurately synchronized with one another by the conversion apparatus <NUM> as previously described. Therefore, performing the multivariate analysis using the plurality of pieces of accurately synchronized sensing data <NUM> enables the learning section <NUM> to extract the sensing data <NUM> highly associated with the evaluation values 506a and to obtain the highly accurate relation information <NUM>.

Furthermore, in the present embodiment, use of the relation information <NUM> indicating the relation between the sensing data <NUM> and the evaluation values 506a is not limited to use as the training data. For example, the server <NUM> inputs the feature variables <NUM> and the evaluation values 506a to the learning section <NUM> as input signals and training signals, respectively. As depicted in <FIG>, the learning section <NUM> performs machine learning on the relation between the feature variables <NUM> and the evaluation values 506a by performing the multivariate analysis such as multiple regression analysis thereon. The learning section <NUM> may then output the information regarding the relation (relation information) <NUM> obtained by the machine learning described above as the training data.

In this way, in the present embodiment, using collective intelligence of the sensing data regarding the plurality of performers or the plurality of trials as the training data makes it possible to extract items of the sensing data <NUM> or the feature variables <NUM> that indicate characteristic tendency in performers at the high performance level and performers at the low performance level. It is noted that the relation information <NUM> described above may be not only used as the training data at the time of performing comparison in a subsequent step but also referred to at the time of selecting one or a plurality of pieces of training data from among a plurality of pieces of training data for the comparison.

It is noted that, at the time of performing the machine learning, the sensing time widths in the trials do not always match one another; thus, it is preferable for the server <NUM> to match the time widths of the sensing data in the trials with one another on the basis of, for example, the timing of touching the keys.

Next, <FIG> depicts details of creation of the comparison data in Step S300 of <FIG> described above. <FIG> is a flowchart of creating the comparison data according to the present embodiment. As depicted in <FIG>, the creation of the comparison data includes a plurality of steps from Step S301 to Step S309. It is noted that Steps S301 to S309 of <FIG> correspond to Steps S201, S203, S207, S209, and S213 of <FIG>; thus, detailed description thereof is omitted herein.

Details of the comparison in Step S400 of <FIG> described above will next be described with reference to <FIG>. <FIG> is a flowchart of the comparison according to the present embodiment and <FIG> and <FIG> are explanatory diagrams of an example of the comparison. As depicted in <FIG>, the comparison according to the present embodiment includes a plurality of steps from Step S401 to Step S411. Details of the steps will be described hereinafter.

First, the server <NUM> selects the sensing data group <NUM> containing the test piece (the piece of music, the phrase, the scale, the arpeggio, the chord, or the like) performed by the learner or part of the test piece as selection of the training data from the DB <NUM>. At this time, the server <NUM> can select the sensing data group <NUM> on the basis of the attribute information associated with the performance contained in the metafile <NUM>. At this time, in the case of presence of a plurality of sensing data groups <NUM> in the DB <NUM>, it is preferable that the server <NUM> creates a list of these plurality of sensing data groups <NUM> and displays the list to the user. The user then selects the sensing data group <NUM> desired to be used for comparison from the list. At this time, the user may select the sensing data group <NUM> related to a performance of a performer other than the learner or may select the sensing data group <NUM> related to a past performance of the learner.

Furthermore, in the case of presence of the plurality of sensing data groups <NUM> in the DB <NUM>, the server <NUM> may extract the sensing data group <NUM> having attribute information regarding a performer similar to the attribute information (the gender, the age, the physical size, the muscle force, the tenderness, the legerity, and the like) regarding the learner according to the attribute information regarding the learner and the like, and recommend the extracted sensing data group <NUM> to the user. Alternatively, the server <NUM> may extract the sensing data group <NUM> having attribute information associated with the sensing and similar to the attribute information associated with the sensing (the sensor type, the sampling frequency, the number of channels, and the like) in the trial of the learner according to the attribute information in the trial of the learner and the like, and recommend the extracted sensing data group <NUM> to the user. In another alternative, the server <NUM> may extract the sensing data group <NUM> having attribute information regarding a trial similar to the attribute information regarding the trial (the competition name, the competition level, and the like) of the learner and attribute information regarding a user's desired trial (the attendance, the number of fans, the evaluation information regarding attendance, the number of views, and the like) according to these pieces of attribute information, and recommend the extracted sensing data group <NUM> to the user. In still another alternative, the server <NUM> may extract the sensing data group <NUM> according to the user's desired sensitivity evaluation and the evaluation values 506a for the performance, and recommend the extracted sensing data group <NUM> to the user. In the present embodiment, by doing so, it is possible to reduce the number of options for the user and for the user to easily select the sensing data group <NUM>.

Next, the user makes a selection as to the feature variables <NUM> regarding what part and what motion (for example, how to use muscles, a motion of the body, and the like) in a body movement are used for the comparison to realize a desired state of the performance (for example, fast piano playing like a famous pianist). In other words, the user selects one or a plurality of pieces of the sensing data <NUM>, the feature variables <NUM>, or the relation information <NUM> for use as the training data from the sensing data group <NUM> selected in Step S401 described above. At this time, there is a probability of presence of vast amounts of data that can be candidates of the training data in the sensing data group <NUM>; thus, the server <NUM> may extract suitable training data and recommend the suitable training data to the user, or automatically select the suitable training data.

For example, the server <NUM> calculates a difference between each sensing data <NUM> in the sensing data group <NUM> selected in Step S401 and the sensing data <NUM> regarding the learner of the same item as that of the former sensing data <NUM>. The server <NUM> then recommends to the user the sensing data <NUM> greatest in difference as the training data. Alternatively, for example, the server <NUM> may automatically select the sensing data <NUM> estimated to be highly related to the user's desired state of the performance (evaluation values 506a) by referring to the relation information <NUM> as the training data. In the present embodiment, by doing so, it is possible to reduce the number of options for the user and for the user to easily select the training data.

Next, the user selects the sensation modality (the visual sensation, the auditory sensation, the tactile sensation, or the like) suited to feed back the feedback information generated in Step S411 to be described later to the user. In the present embodiment, the feedback information can be fed back, for example, by the visual sensation via the display apparatus, by the auditory sensation via the audio output apparatus, or by the tactile sensation via the wearable apparatus (haptic mechanism) attached to the body of the learner. Furthermore, in the present embodiment, the user may select a plurality of sensation modalities.

Next, the server <NUM> performs comparison either by calculating the difference between the training data selected in Step S403 and the sensing data <NUM> (comparison data) regarding the learner of the item same as that of the training data, or by superimposing the training data on the comparison data. As depicted in, for example, <FIG>, the comparison section <NUM> of the server <NUM> compares training data <NUM> with comparison data <NUM>. In subsequent Step S411, feedback information <NUM> is then generated on the basis of a comparison result. It is noted that, in Step S407, the server <NUM> may calculate the difference between the selected training data <NUM> and the comparison data <NUM> as previously described, or superimpose the training data <NUM> on the comparison data <NUM> as depicted in <FIG>.

Next, the server <NUM> may emphasize or reduce the difference between the training data and the comparison data obtained in Step S407 described above by expanding or contracting the difference either spatially or temporally. Specifically, in the present embodiment, the difference is reduced by a predetermined contraction ratio for the learner at the skill level greater in difference from that in the training data <NUM> to avoid the decrease of motivation of learning. On the assumption, for example, that the key touch force in the training data <NUM> is <NUM>, in a case in which the key touch force of the learner (comparison data) is approximately <NUM>% lower than that in the training data, the server <NUM> provides the difference in key touch force contracted (reduced) to <NUM>% without providing the difference kept at <NUM>%. On the other hand, in the present embodiment, the small difference is expanded by a predetermined magnification for the learner at the skill level smaller in difference from that in the training data <NUM> to guide the learner to a higher skill level in such a manner that the learner can easily recognize the difference. For example, in a case in which the key touch timing of the learner (comparison data) is <NUM> seconds faster than that in the training data <NUM>, the server <NUM> provides the difference expanded (emphasized) to <NUM> seconds without providing the difference kept at <NUM> seconds in such a manner that the learner can easily recognize the difference. In other words, in the present embodiment, performing the processing on the difference described above makes it possible for the learner to easily recognize the difference from the training data <NUM> while avoiding the decrease of the motivation of the learner; thus, it is possible to realize assistance of efficient learning of the performance. It is noted that the magnification or the contraction ratio for the difference may be adjusted step by step according to the change in the skill level of the learner in the present embodiment. Furthermore, temporal expansion or contraction means slowing down or speeding up the time associated with providing of the feedback information.

The server <NUM> generates the feedback information <NUM> to be fed back to the user according to the processing from Step S401 to S409 described above.

As described so far, according to the embodiment of the present disclosure described above, it is possible to provide the information processing system <NUM> capable of accurately synchronizing and handling the sensing data in a mixture of forms and available for assisting in learning of the performance.

Meanwhile, in the embodiment of the present disclosure described above, it is preferable to provide a learning assistance mode to be described hereinafter such that effectiveness of the learning assistance based on the feedback provided by the information processing system <NUM> described above can be further enhanced. Therefore, the learning assistance mode according to the embodiment of the present disclosure will be described hereinafter with reference to <FIG> is a flowchart of the learning assistance mode according to the present embodiment. It is assumed that the user can select in advance whether to execute the learning assistance mode.

Meanwhile, as elements associated with contents of the feedback information <NUM> among elements of the feedback that have an influence on the learning of the learner, items of the training data <NUM> used in the comparison (what sensing data <NUM> and what feature variables <NUM> are to be selected), a method of the comparison (whether the difference between the training data <NUM> and the comparison data <NUM> is calculated or these pieces of data are superimposed), and the like can be cited. Furthermore, as elements associated with output of the feedback information <NUM>, a degree of emphasis or reduction of the difference, the sensation modality by which the feedback information is fed back, and the like can be cited. Which of these elements is more effective for the learning assistance in what way varies depending on the performance to be learned and characteristics and circumstances of the learner. In the acquisition assistance mode according to the present embodiment, therefore, which of the elements is more effective for the learning assistance in what way is searched in real time and higher effective feedback is provided.

For example, in a case in which the learner or the like is unable to realize a desired performance, a plurality of elements (feature variables) is often present as causes. In the case, for example, of a mistake in key touch during the performance, a plurality of elements that causes the mistake in the key touch is present including wrong forms of learner's arms, tensing up of the muscles, and delayed timing of looking at the keys to be touched. Therefore, an algorithm that can be obtained by the learning assistance mode makes it possible to select such an element that "the learner will be able to avoid making mistakes in the key touch at the earliest if correcting this element. " For example, the algorithm selects the element for which the difference between the comparison data and the training data is the greatest among the plurality of factors (feature variables), or selects the feature variables said to be important according to experiences of many performers. Using such an algorithm enables the learning assistance to be performed more effectively.

For example, in the learning assistance mode, the learner is made to repeat trials of the same test piece approximately a few times (for example, three to five times), and the server <NUM> repeats acquisition and analysis of the sensing data group <NUM> and presentation of the feedback information <NUM>.

At this time, the server <NUM> extracts the evaluation values 506a (state of the performance) for the performance of the learner from the sensing data <NUM>, and calculates changes in evaluation values 506a from those in the previous trial. At this time, in a case in which it is not confirmed that the evaluation values 506a have improved, the server <NUM> changes the items of the training data <NUM> used in the comparison, the method of the comparison, the degree of the emphasis or the reduction of the difference, the sensation modality, and the like, and feeds back the changes before a next trial. On the other hand, in a case in which it is confirmed that the evaluation values 506a have improved, the server <NUM> feeds back the feedback information before the next trial without changing the training data <NUM> used in the comparison, the method of the comparison, the degree of the emphasis or the reduction of the difference, the sensation modality, and the like.

Furthermore, repeatedly carrying out such trials and feedback enables the learning section <NUM> of the server <NUM> to perform the machine learning on a relation of the evaluation values 506a with the items of the training data <NUM>, the method of the comparison, the degree of the emphasis or the reduction of the difference, and the sensation modality. In other words, the learning section <NUM> can make it clear, by the machine learning, the items of the training data <NUM> used in the comparison, the method of the comparison, the degree of the emphasis or the reduction of the difference, the sensation modality, and the like suited for improving a state of a specific performance of the learner. Furthermore, the server <NUM> can select the suited items of the training data <NUM>, the suited method of the comparison, the suited degree of the emphasis or the reduction of the difference, and the suited sensation modality using the algorithm based on the relation obtained by such machine learning, and recommend a selection result to the user. Moreover, the server <NUM> may apply the algorithm obtained in this way to another learner. For example, it is estimated that a learner, if similar in attribute information (the name, the gender, the age, the body height, the body weight, the physical size, the muscle force, the palm size, and the like), tends to have similar items of the training data <NUM> used in the comparison, a similar method of the comparison, a similar degree of the emphasis or the reduction of the difference, and a similar sensation modality suited for improving the state of the specific performance; thus, the server <NUM> may determine whether to apply the algorithm to the other learner on the basis of the attribute information regarding the other learner.

Furthermore, in the learning assistance mode, in a case in which there appears to be a correlation between an improvement of a motility function (the muscle force, joint ranges of motion, quickness, and sensory functions) of the learner and the improvement of the state of the specific performance, the server <NUM> preferably not only performs the feedback but also recommends a training for improving the motility function. For example, it is known that a performer ununiform in rhythm and incorrect in rhythm during the performance is lower in a tactile function of fingertips and lower in a function to independently move fingers. The server <NUM>, therefore, recommends to the learner, as the training for improving the motility function, a finger independent moving exercise and a tactile function training (sensory learning training) in a case in which the learner aims to improve a performance, that is, "to improve strictness.

More specifically, as depicted in <FIG>, the learning assistance mode according to the present embodiment includes a plurality of steps from Step S601 to Step S609. Details of these steps will be described hereinafter.

The feedback apparatus <NUM> performs feedback toward the learner on the basis of the feedback information <NUM> received from the server <NUM>, similarly to the above description.

The learner executes a trial of the same test piece as that in a previous trial while referring to the feedback. The server <NUM> acquires the sensing data group <NUM> related to the executed trial.

The server <NUM> extracts the evaluation values 506a for the performance of the learner on the basis of the sensing data group <NUM> acquired in Step S603 described above. Furthermore, the server <NUM> calculates changes in the evaluation values 506a from those in the previous trial.

In the case in which it is not confirmed that the evaluation values 506a have improved, the server <NUM> selects to change the items of the training data <NUM> used in the comparison, the method of the comparison, the degree of the emphasis or the reduction of the difference, the sensation modality, and the like. On the other hand, in the case in which it is confirmed that the evaluation values 506a have improved, the server <NUM> selects not to change the items of the training data <NUM> used in the comparison, the method of the comparison, the degree of the emphasis or the reduction of the difference, the sensation modality, and the like.

At the time of Steps S605 and S607 described above, the server <NUM> performs the machine learning on the relation of the evaluation values 506a with the items of the training data <NUM>, the method of the comparison, the degree of the emphasis or the reduction of the difference, and the sensation modality. As previously described, the server <NUM> may perform next feedback using the algorithm based on the relation obtained by the machine learning.

Next, the server <NUM> generates the feedback information <NUM> on the basis of selection in Step S607 described above. Furthermore, the server <NUM> selects a training for improving the motility function to be recommended to the learner.

The embodiment of the present disclosure has been described so far. Examples of application of the embodiment of the present disclosure will next be described more specifically. It is noted that the following examples of application are given as an example of the embodiment of the present disclosure only, and the embodiment of the present disclosure is not limited to the following examples of application.

As one example of application of the embodiment of the present disclosure, skill acquisition assistance of sports, arts (paintings, calligraphy, and the like), performing arts and traditional arts, various operation simulators (for motor vehicles, airplanes, and the like), and games can be cited besides the learning of musical instrument playing techniques. For example, attaching the sensor apparatus <NUM> described above to a tennis racket or the like enables identification of a skill element necessary to improve in a tennis performance of the user and automatic coaching to the user and the like.

Furthermore, as examples of application of the embodiment of the present disclosure, assistance of diagnosis of a patient affected with an impaired motility function, evaluation of effectiveness of treatment, and prediction assistance of a future impaired motility function can be cited. While it is known, for example, on a job site of music therapy that practice of musical instrument playing contributes to a recovery from a motility function or a cognitive function after a cerebral stroke, most of a mechanism of the recovery is not reached yet. In the present embodiment, therefore, it is also possible to accurately synchronize data associated with a brain activity of a patient with data associated with a movement such as the musical instrument playing, and acquire these pieces of data; thus, it is expected to acquire information beneficial for making the mechanism clear.

Moreover, the DB <NUM> constructed in the present embodiment is also beneficial for development of various types of coaching methods for skill learning; thus, the DB <NUM> could be a product to be traded solely.

<FIG> is an explanatory diagram illustrating an example of a hardware configuration of an information processing apparatus <NUM> according to the present embodiment. In <FIG>, the information processing apparatus <NUM> indicates an example of the hardware configuration of the server <NUM> described above.

The information processing apparatus <NUM> has, for example, a CPU <NUM>, a ROM <NUM>, a RAM <NUM>, a recording medium <NUM>, and an input/output interface <NUM>. The information processing apparatus <NUM> further has a haptic device <NUM>, a display device <NUM>, an audio output device <NUM>, a communication interface <NUM>, and a sensor <NUM>. Moreover, the information processing apparatus <NUM> connects the constituent elements to one another by, for example, a bus <NUM> that serves as a data transmission passage.

The CPU <NUM> is configured with, for example, one or two or more processors each including a computing circuit such as a CPU, every type of processing circuit, and the like, and functions as the main control section <NUM> that exercises control over the entire information processing apparatus <NUM>.

The ROM <NUM> stores therein programs, control data such as computing parameters, and the like used by the CPU <NUM>. The RAM <NUM> temporarily stores therein the programs and the like executed by the CPU <NUM>. The ROM <NUM> and the RAM <NUM> function as, for example, the storage section <NUM> described above in the information processing apparatus <NUM>.

The recording medium <NUM> functions as the storage section <NUM> described above, and stores, for example, data related to the information processing method according to the present embodiment, and various types of data such as various applications. Examples of the recording medium <NUM> include herein a magnetic recording medium such as a hard disk and a nonvolatile memory such as a flash memory. Furthermore, the recording medium <NUM> may be detachable from the information processing apparatus <NUM>.

The input/output interface <NUM> connects, for example, the haptic device <NUM>, the display device <NUM>, the audio output device <NUM>, and the like. Examples of the input/output interface <NUM> include a USB (Universal Serial Bus) terminal, a DVI (Digital Visual Interface) terminal, an HDMI (High-Definition Multimedia Interface) (registered trademark) terminal, and various types of processing circuits.

The haptic device <NUM> functions as the haptic mechanism <NUM> described above, the display device <NUM> functions as the display section <NUM> described above, and the audio output device <NUM> functions as the audio output section <NUM> described above. As the haptic device <NUM>, a wearable device attached to the body of the learner can be cited. As the audio output device <NUM>, a speaker, a headphone speaker, and the like can be cited. Furthermore, as the display device <NUM>, a liquid crystal display, an organic EL display (Organic ElectroLuminescence Display), and the like can be cited.

Needless to say, the input/output interface <NUM> can be connected to an external device such as an operation input device (for example, a keyboard and a mouse) outside of the information processing apparatus <NUM> and an external display device.

The communication interface <NUM> is communication means that functions as the communication section <NUM> and that is provided in the information processing apparatus <NUM>, and functions as a communication section (not depicted) for holding either wireless or wired communication with an external apparatus via the network <NUM> (or directly). Examples of the communication interface <NUM> include a communication antenna, an RF (Radio Frequency) circuit (for wireless communication), an IEEE802. <NUM> port and a transmitting-receiving circuit (for wireless communication), an IEEE802. <NUM> port and a transmitting-receiving circuit (for wireless communication), and a LAN (Local Area Network) terminal and a transmitting-receiving circuit (for wired communication).

The example of the hardware configuration of the information processing apparatus <NUM> has been illustrated so far. It is noted that the hardware configuration of the information processing apparatus <NUM> is not limited to the configuration depicted in <FIG>. More specifically, the above constituent elements may be configured using general-purpose members, or configured by hardware dedicated to functions of the constituent elements. Such a configuration could be changed as appropriate according to technology levels at different times of carrying out the present disclosure.

For example, the information processing apparatus <NUM> is not necessarily configured with the communication interface <NUM> in the case of communication with an external apparatus or the like via an external communication device connected to the information processing apparatus <NUM> or in the case of being configured to perform processing in a stand-alone fashion. Furthermore, the communication interface <NUM> may be configured to be capable of communication with one or two or more external apparatuses by a plurality of communication schemes.

Moreover, the information processing apparatus according to the present embodiment may be applied to a system configured from a plurality of apparatuses on the premise of connection to a network such as cloud computing (or of communication between the apparatuses). In other words, the information processing apparatus according to the present embodiment described above can be realized as, for example, an information processing system performing processing related to the information processing method according to the present embodiment using the plurality of apparatuses.

It is noted that the embodiment of the present disclosure described previously could include, for example, a program for causing a computer to function as the information processing apparatus (for example, the server <NUM>) according to the present embodiment, and a non-transitory tangible medium recording therein a program. Furthermore, the program may be distributed via a communication line (including a line for wireless communication) such as the Internet.

Furthermore, the steps in each processing in the embodiment of the present disclosure described above are not necessarily processed in a described order. For example, the steps may be processed by changing the order as appropriate. Moreover, the steps may be partially processed either in parallel or individually as an alternative to be processed in time series. Furthermore, as for a method of processing the steps, the steps are not necessarily processed in accordance with the described method. For example, the steps may be processed by other methods by other functional sections.

While the preferred embodiment of the present disclosure has been described in detail with reference to the accompanying drawings, a technical scope of the present disclosure is not limited to such an example.

Claim 1:
An information processing apparatus (<NUM>) comprising:
a conversion section (<NUM>) that converts sensing data in different forms obtained from each of a plurality of sensors (<NUM>) each sensing a state related to a performance by a motion of a user;
an information processing section (<NUM>) that processes the sensing data converted by the conversion section (<NUM>), wherein processing of the sensing data includes extracting feature variables characterizing a state of the performance; and
an information output section (<NUM>) that outputs feedback information to the user on a basis of a processing result of the information processing section (<NUM>), wherein
the conversion section (<NUM>) includes
an analog-digital signal conversion section (<NUM>) that converts sensing data acquired from a first sensor (10a), that outputs sensing data in an analog form, into sensing data in a digital form and outputs the sensing data in the digital form to the information processing section (<NUM>), wherein
the analog-digital signal conversion section (<NUM>) has a first clock mechanism for synchronizing sensing data in the analog form from the first sensor (10a) that outputs the sensing data in the analog form, and
a digital-analog signal conversion section (<NUM>) that converts sensing data acquired from a second sensor (10b), that outputs sensing data in the digital form, into sensing data in the analog form and outputs the sensing data in the analog form to the analog-digital signal conversion section (<NUM>), wherein
the information processing apparatus (<NUM>) further comprises:
a second clock mechanism that is provided closer to the second sensor (10b) that outputs sensing data in the digital form for synchronizing the sensing data from the second sensor (10b), and
the information processing section (<NUM>) synchronizes the sensing data from the first sensor (10a) with the sensing data from the second sensor (10b) on a basis of a temporal difference between the first clock mechanism and the second clock mechanism.