An information processing apparatus including an acquisition section that acquires an operation model indicating an operation pattern related to a target object, a calculation section that calculates an overall operation including an identified partial operation of operations of the target object, with reference to the operation model acquired by the acquisition section, and an output control section that outputs information indicating the overall operation of the target object that has been calculated by the calculation section.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a U.S. National Phase of International Patent Application No. PCT/JP2017/010517 filed on Mar. 15 2017, which claims priority benefit of Japanese Patent Application No. JP 2016-119688 filed in the Japan Patent Office on Jun. 16, 2016. Each of the above-referenced applications is hereby incorporated herein by reference in its entirety.

TECHNICAL FIELD

The present disclosure relates to an information processing apparatus, an information processing method, and a storage medium.

BACKGROUND ART

Recently, technology to visualize (that is, to digitize) a move of a body has been actively developed. In the field of sports, for example, technology is being developed to attach sensor devices to various parts of a body for visualizing a move of the body on the basis of the measurement results and contributing to improvement in forms, and the like.

For example, Patent Literature 1 described below discloses a technology of appropriately setting an analysis section of output data obtained from a sensor device mounted on a user or a tool used by a user, for enhancing determination accuracy of a motion pattern such as a serve and a smash in tennis.

CITATION LIST

Patent Literature

DISCLOSURE OF INVENTION

Technical Problem

Nevertheless, it is hard to say that the technology proposed in Patent Literature 1 described above is sufficient as a technology to visualize an operation of a target object. For example, one aspect of the insufficiency lies on the reduction of an information amount required for visualizing the operation of the target object. In view of the foregoing, a mechanism that can visualize an operation of a target object from a smaller amount of information is desirably provided.

Solution to Problem

According to the present disclosure, there is provided an information processing apparatus including: an acquisition section configured to acquire an operation model indicating an operation pattern related to a target object; a calculation section configured to calculate an overall operation including an identified partial operation of operations of the target object, with reference to the operation model acquired by the acquisition section; and an output control section configured to output output information indicating the overall operation of the target object that has been calculated by the calculation section.

In addition, according to the present disclosure, there is provided an information processing method including: acquiring an operation model indicating an operation pattern related to a target object; calculating, by a processor, an overall operation including an identified partial operation of operations of the target object, with reference to the acquired operation model; and outputting output information indicating the calculated overall operation of the target object.

In addition, according to the present disclosure, there is provided a storage medium storing a program for causing a computer to function as: an acquisition section configured to acquire an operation model indicating an operation pattern related to a target object; a calculation section configured to calculate an overall operation including an identified partial operation of operations of the target object, with reference to the operation model acquired by the acquisition section; and an output control section configured to output output information indicating the overall operation of the target object that has been calculated by the calculation section.

Advantageous Effects of Invention

As described above, according to the present disclosure, a mechanism that can visualize an operation of a target object from a smaller amount of information is provided. Note that the effects described above are not necessarily limitative. With or in the place of the above effects, there may be achieved any one of the effects described in this specification or other effects that may be grasped from this specification.

MODE(S) FOR CARRYING OUT THE INVENTION

In addition, in the specification and the drawings, different alphabetical letters may be given to components having substantially the same functional configuration for distinction after the same symbol is given to the components. For example, a plurality of components having substantially the same functional configuration are distinguished as sensor devices10A,10B, and10C as necessary. However, in a case where it is unnecessary to particularly distinguish each of the plurality of components having substantially the same functional configuration, only the same symbol is given. For example, in a case where it is unnecessary to particularly distinguish sensor devices10A,10B, and10C, the sensor devices are simply referred to as a sensor device10.

1. First Embodiment1.1. Outline of system1.2. Configuration example of sensor device1.3. Configuration example of information processing apparatus1.4. Technical feature1.5. Flow of processing

2. Second Embodiment2.1. Configuration example of information processing apparatus2.2. Technical feature2.3. Flow of processing

3. Hardware configuration example

1. First Embodiment

The present embodiment is a mode of visualizing an operation of a real object.

FIG. 1is a diagram for describing an outline of a system1according to the present embodiment. As illustrated inFIG. 1, the system1includes a plurality of sensor devices10(that is,10A to10C) attached to a sensor attachment apparatus20.

The sensor device10is a device that measures various kinds of data. The sensor device10is attached to a sensor attachment tool21included in the sensor attachment apparatus20to perform measuring targeting a move of a target object. A target object may be a human, a dog, a cat, or other living organisms, or may be a non-living organism such as a robot. In the example illustrated inFIG. 1, a target object is a user (that is, a human). In addition, the target object may be an object to be used by a living organism. For example, the target object may be a tool to be used for games such as a golf club, a tennis racket, a ski board, a ski boot, a goal, or a bat. In addition, the target object may be a tool to be used for living such as an artificial hand or a wheelchair. In addition, the target object may be a tool to be used for animals such as a collar or a horseshoe.

The sensor device10transmits information indicating a measurement result (hereinafter, also referred to as sensor information), to an information processing apparatus30. The transmission may be performed in real time concurrently with the measurement, or the information may be stored and transmitted at an arbitrary timing after the measurement.

The sensor attachment apparatus20is an apparatus for fixing the sensor device10to a target object. As illustrated inFIG. 1, the sensor attachment apparatus20has one or more attachment positions (the sensor attachment tool21) for removably attaching the sensor devices10, and the sensor devices10can be attached to a part of or all of the attachment positions. The sensor attachment apparatus20may be formed into a shape that covers a part of or all of the trunk, the limbs, or the like of a user, and in that case, it is desirable to form the sensor attachment apparatus20with extendable and retractable materials so that a move of a user is not disturbed. In addition, the attached sensor device10may be separated from the target object, and the sensor attachment apparatus20may have thickness like a helmet, a protector, and the like do. Additionally, the sensor attachment apparatus20may be attached to or be integrally formed with an object such as a golf club, a tennis racket, and a ski board. A user can attach the sensor device10to the sensor attachment tool21positioned in a region that the user wants to measure.

The information processing apparatus30acquires the sensor information from the sensor device10, and performs various kinds of processing for visualizing an operation of a target object.

<1.2. Configuration Example of Sensor Device>

FIG. 2is a block diagram illustrating an example of a configuration of the sensor device10according to the present embodiment. As illustrated inFIG. 2, the sensor device10according to the present embodiment includes an inertial sensor110, a communication section120, a storage section130, and a control section140.

The inertial sensor110is a device that performs measurement using inertia. The inertial sensor110includes an acceleration sensor, a gyro sensor, a geomagnetic sensor, and the like, and outputs the measured sensor information (e.g. acceleration and angular speed) to the control section140.

The communication section120is a communication module for performing transmission and reception of data between itself and the information processing apparatus30in a wired/wireless manner. The communication section120can perform communication conforming to an arbitrary communication method such as a Local Area Network (LAN), a wireless LAN, Wi-Fi (registered trademark), Bluetooth (registered trademark), or infrared communication, for example. The communication section120transmits the sensor information measured by the inertial sensor110, to the information processing apparatus30.

The storage section130temporarily or permanently stores programs and various types of data for operations of the sensor device10. For example, the storage section130temporarily stores the information measured by the inertial sensor110.

The control section140corresponds to a Central Processing Unit (CPU), a Digital Signal Processor (DSP), or the like, and performs processing for providing various types of functions of the sensor device10. The sensor device10operates on the basis of control performed by the control section140. The operation of the sensor device10that is based on the control performed by the control section140will be described in detail later.

<1.3. Configuration Example of Information Processing Apparatus>

FIG. 3is a block diagram illustrating an example of a configuration of the information processing apparatus30according to the present embodiment. As illustrated inFIG. 3, the information processing apparatus30according to the present embodiment includes a communication section310, an input section320, an output section330, a storage section340, and a control section350.

The communication section310is a communication module for performing transmission and reception of data between itself and the sensor device10in a wired/wireless manner. The communication section310can perform communication conforming to an arbitrary communication method such as a LAN, a wireless LAN, Wi-Fi, Bluetooth, or infrared communication, for example. The communication section310receives the sensor information from the sensor device10.

The input section320receives an input of information. For example, the input section320receives an input of information from the user. The input section320outputs the input information to the control section350.

The output section330performs an output of information. For example, the output section330outputs information using an image, a sound, vibration, and/or the like. The output section330outputs information on the basis of control performed by the control section350.

The storage section340temporarily or permanently stores programs and various types of data for operations of the information processing apparatus30. For example, the storage section340stores an operation model to be described later.

The control section350corresponds to a CPU, a DSP, or the like, and performs processing for providing various types of functions of the information processing apparatus30. The control section350may be regarded as at least one electrical circuit formed so as to be able to execute functional units disclosed inFIG. 3. As illustrated inFIG. 3, the control section350includes a model learning section351, a model acquisition section352, a first position information calculation section353, a second position information calculation section354, and an output control section355. Note that the control section350can further include structural elements other than these structural elements. In other words, the control section350can also perform operations other than operations of these structural elements. The operations of these structural elements will be described in detail later.

(1) Target Object

In the present embodiment, a target object is a real object. In addition, position information in the present embodiment is a three-dimensional coordinate (i.e. X-coordinate, Y-coordinate, and Z-coordinate) in a real space.

For example, the target object includes one or more moving objects. In other words, the target object may be singular or plural. For example, a human and an object (e.g. tool) manipulated by a human can be the target object. The operation of the target object is represented by a time series variation of the attitude of the target object.

For example, the target object may include a moving object having a plurality of joints. Examples of such a target object include a human, a robot, and the like. In addition, the operation of the target object may include time series variations of position information pieces of the plurality of joints of the moving object. In other words, the information processing apparatus30can visualize a complicated operation in which a positional relationship between the joints varies.

In the present embodiment, a target object to be visualized is assumed to be a human. Hereinafter, a human serving as a target object will also be referred to as a user.

The sensor device10is arranged in an arbitrary region of the target object. For example, the sensor device10may be arranged at a joint of the user. In this case, the sensor attachment tool21is arranged at the joint of the user. A region in which the sensor device10is to be arranged, such as a position of the sensor attachment tool21to which the sensor device10is to be attached, for example, will also be referred to as a first point. On the other hand, a region in which the sensor device10is not arranged, such as a position of the sensor attachment tool21to which the sensor device10is not attached, for example, will also be referred to as a second point.

A manipulator of the information processing apparatus30and the user may be the same person, or may be different persons. As an example, the following description will be given assuming that the manipulator and the user are the same person.

(2) Calculation of Position Information that is Based on Sensor Information

On the basis of sensor information measured by one or more sensor devices10arranged on the target object, the information processing apparatus30(e.g. the first position information calculation section353) identifies a time series variation of position information of the one or more sensor devices10. More briefly, the information processing apparatus30calculates position information of the first point on the basis of the sensor information. For example, the information processing apparatus30calculates the position information of the first point from the sensor information acquired from the sensor device10, using an inertial navigation system (INS).

The inertial navigation system is a technology of calculating a sensor position by integrating angular speed and acceleration a plurality of times, and is employed in ships, airplanes, or the like, for example. In the inertial navigation system, first of all, by integrating angular speed (performing first integration), an attitude (i.e. an attitude angle in the real space) of a sensor device is calculated. Subsequently, by integrating acceleration (performing second integration), speed of the sensor device is calculated. Next, by integrating speed (performing third integration), a moving distance of the sensor device is calculated. Then, by combining vectors of moving distances and attitudes (i.e. moving directions) for each subdivision point, relative position information starting from an initial position is calculated. If the initial position is already known, absolute position information (i.e. a three-dimensional coordinate in the real space) of the sensor device can be calculated by the above calculation.

For example, an example of measuring a state in which the user swings a golf club called an iron will be assumed. As an example, the sensor devices10are assumed to be attached to a neck, an waist, a right knee, a right foot, a left knee, a left foot, a hand, and a club head of the iron. In this case, the information processing apparatus30can visualize an operation in which the user swings the iron, by calculating position information of each of the sensor devices10. The example of the visualization will be described with reference toFIG. 4.

FIG. 4is a diagram illustrating an example of visualization processing of a user operation according to the present embodiment. Attitudes40A to40F indicate attitudes (i.e. position information pieces of the respective sensor devices10) at the respective timings of the user that swings the iron, and time flows in order from the attitude40A toward the attitude40F. The attitude40A indicates an attitude at a timing at which the user holds the iron at the ready. The attitude40B indicates an attitude at a timing of a back swing. The attitude40C indicates an attitude at a timing at which the back swing has reached the top. The attitude40D indicates an attitude at a timing of a down swing. The attitude40E indicates an attitude at a timing of follow through. The attitude40F indicates an attitude at a timing of finish. Plots41in each attitude indicate positions to which the sensor devices10are attached. A plot41A corresponds to a neck, a plot41B corresponds to an waist, a plot41C corresponds to a right knee, a plot41D corresponds to a right foot, a plot41E corresponds to a left knee, a plot41F corresponds to a left foot, a plot41G corresponds to a hand, and a plot41H corresponds to a club head of the iron.

Aside from the inertial navigation system, the information processing apparatus30can calculate the position information of the first point using an arbitrary algorithm. For example, by an optical motion capture technology that uses a captured image of a marker provided on a partial joint of the user, the information processing apparatus30may calculate position information of the joint to which the marker is added.

(3) Operation Model

An operation model is information indicating an operation pattern of a modeled object related to a target object.

The modeled object is an object to which the sensor device10is attached. The target object and the modeled object may be the same, or may be different. For example, the user and a person modeled as a modeled object may be the same person, or may be different persons.

The operation model differs depending on the context, and is stored in the storage section340for each context. In other words, the operation model can include a plurality of operation models corresponding to contexts, such as a first operation model corresponding to a first context, and a second operation model corresponding to a second context. The operation model is information indicating a time series variation of position information of each region of a modeled object that is obtainable in a case where the information processing apparatus30operates in a context in which there is a modeled object. As an example, an operation model related to a swing operation of the iron will be described. In this case, the operation model includes information indicating time series variations of position information pieces of the respective regions of the neck, the waist, the right knee, the right foot, the left knee, the left foot, the hand, and the club head of the iron, for example. Here, the respective regions corresponding to the position information pieces included in the operation model correspond to regions in which the sensor devices10are arranged (i.e. positions in which the sensor attachment tool21are arranged).

Various types of contexts can be considered. For example, the contexts can include the type of the modeled object, the type of an operation performed by the modeled object, attribute information of the modeled object, and information indicating a state. Specifically, for example, in a case where the modeled object is a person, the contexts can include the type of an operation such as walking, running, golf, and tennis, attribute information such as gender, age, a body height, and a body weight, information indicating a health state, a habit of an operation, etc., and the like.

For example, the operation model can be represented as a regression model. In addition, the operation model may be represented in a format obtained by dimension-compressing multidimensional information.

In addition, the information processing apparatus30(the model learning section351) may learn an operation model. For example, as for a swing operation of the iron, the information processing apparatus30generates an operation model on the basis of calculation of position information that is based on sensor information obtainable in a case where the user performs a swing operation in a state in which the sensor devices10are attached to all the sensor attachment tools21and the club head of the iron.

(4) Selection of Operation Model

The information processing apparatus30(e.g. the model acquisition section352) acquires at least one operation model related to a target object. Specifically, the information processing apparatus30acquires, from among operation models stored in the storage section340, an operation model corresponding to the context of the target object. For example, as for a swing operation of the iron, the information processing apparatus30acquires an operation model in which a modeled object is a person, and the type of an operation is golf.

For example, the information processing apparatus30may acquire an operation model on the basis of a user manipulation. For example, the context of the user is input to the information processing apparatus30by the user itself, and an operation model corresponding to the input context is selected. In this case, it becomes possible to correctly refer to an operation model corresponding to the context designated by the user. Note that the user manipulation includes both of an intentionally-performed manipulation (e.g. manual manipulation) and an unconsciously-performed manipulation (e.g. image recognition of an operation).

For example, the information processing apparatus30may acquire an operation model corresponding to an identified partial operation of operations of the target object. Therefore, first of all, the information processing apparatus30recognizes a context on the basis of a partial operation of the user that has been identified using the sensor information measured by the sensor device10arranged on the user. In other words, the information processing apparatus30recognizes a context of the user on the basis of the time series variation of the calculated position information of the first point. Subsequently, the information processing apparatus30automatically acquires, from the storage section340, an operation model corresponding to the recognized context. For example, the information processing apparatus30automatically acquires one of the first operation model and the second operation model in accordance with the recognized context. In this case, it becomes possible for the information processing apparatus30to reduce the burden of a context input.

Note that the information processing apparatus30may acquire the first operation model and the second operation model at different timings. In this case, the information processing apparatus30can change an operation model to be referred to, in accordance with a change in the context of the user. In addition, the information processing apparatus30may simultaneously acquire the first operation model and the second operation model. In this case, it becomes possible for the information processing apparatus30to calculate an operation of the target object by combining a plurality of operation models.

(5) Calculation of Position Information that is Based on Operation Model

The information processing apparatus30(e.g. the second position information calculation section354) calculates an overall operation including the identified partial operation of the operations of the target object, with reference to the acquired operation model. Because the information processing apparatus30can calculate the overall operation even in a case where only a part of the operations of the target object is identified, the operation of the target object can be visualized from a smaller amount of information. Note that, as the number of first points increases, the accuracy of visualization is enhanced.

More specifically, the information processing apparatus30calculates a time series variation of position information of another region of the target object that follows an operation pattern indicated by an operation model, and corresponds to a time series variation of position information of one or more sensor devices. In other words, the information processing apparatus30calculates a time series variation of position information of the second point that follows an operation pattern indicated by an operation model, and corresponds to a time series variation of position information of the first point. It thereby becomes possible for the information processing apparatus30to calculate an overall operation of the user that includes time series variations of the position information of the first point and the position information of the second point. Note that, hereinafter, calculating the position information of the second point will also be referred to as predicting.

For example, the information processing apparatus30may calculate an overall operation on the basis of an identified partial operation of the operations of the user, with reference to an operation model. The target object may be a plurality of objects, and the information processing apparatus30may calculate operations of the objects manipulated by the user, on the basis of a part or all of operations of the user that have been identified, with reference to an operation model. For example, the information processing apparatus30calculates an operation of a golf club swung by the user, on the basis of an identified swing operation of the user, with reference to an operation model related to a swing operation of a golf club. It thereby becomes possible to calculate the trajectory of the golf club even if the sensor device10is not attached to the golf club.

The following assumption will be given of an example of measuring a state in which the user swings an iron, similarly to the example illustrated inFIG. 4. Nevertheless, unlike the example illustrated inFIG. 4, the sensor devices10are assumed to be attached only to the right knee and the hand. The example of the visualization will be described with reference toFIG. 5andFIG. 6.

FIG. 5is a diagram illustrating an example of visualization processing of a user operation according to the present embodiment. First of all, the information processing apparatus30(e.g. the first position information calculation section353) calculates position information pieces of first points (i.e. the right knee and the hand) to which the sensor devices10are attached. An attitude42indicates a part of attitudes of the user that is measured at a timing of a back swing, and a plot43C corresponds to the right knee and a plot43G corresponds to the hand. Subsequently, the information processing apparatus30(e.g. the model acquisition section352) selects, as an operation model corresponding to a context of the user, a golf operation model44B from among a walking operation model44A, the golf operation model44B, and a tennis operation model44C. Then, the information processing apparatus30(e.g. the second position information calculation section354) predicts position information pieces of second points (i.e. the neck, the waist, the right foot, the left knee, the left foot, and the club head of the iron), from the position information pieces of the first points, with reference to the golf operation model44B. An attitude45indicates an attitude obtained by adding prediction results of other attitudes to the part of the attitudes of the user that is measured at the timing of the back swing. A plot43A indicates the neck, a plot43B indicates the waist, a plot43D indicates the right foot, a plot43E indicates the left knee, a plot43F indicates the left foot, and a plot43H indicates the club head of the iron.

By performing the above-described prediction processing at each timing of the swing, the information processing apparatus30can visualize the entire operation in which the user swings the iron, as illustrated inFIG. 6.

FIG. 6is a diagram illustrating an example of visualization processing of a user operation according to the present embodiment. Attitudes46A to46F indicate positions (plots43C and43G) of the sensor devices10, and prediction results (plots43A,43B,43D,43E,43F, and43H) of positions of the other regions, at the respective timings of the user that performs a swing operation. Time flows in order from the attitude46A toward the attitude46F, and the timings from the attitude46A to the attitude46F are similar to the respective timings from the attitude40A to the attitude40F inFIG. 4. In addition, the attitude45illustrated inFIG. 5corresponds to the attitude46B inFIG. 6.

In this manner, by predicting position information pieces of the regions to which the sensor devices10are not attached, from the position information pieces of the sensor devices10, the information processing apparatus30can visualize the entire swing operation of the user.

Here, it is desirable that the sensor device10is arranged at a joint corresponding to a context, among a plurality of predefined joints. This is because the arrangement position that can assure prediction accuracy can vary depending on the context. More specifically, as described later, it is desirable that the sensor device10is arranged at a joint corresponding to a factor having a large factor loading that is indicated by a principal component result in each context. For example, in a case where a context is a swing operation of a golf club, it is desirable that the sensor devices10are arranged at two locations including the hand and the knee (the right knee in the case of right handedness). It thereby becomes possible to predict a swing operation using a smaller number of sensor devices10.

Various types of prediction algorithms of position information pieces of second points can be considered.

Linear Regression

For example, using a linear regression model, the information processing apparatus30(e.g. the second position information calculation section354) may predict position information pieces of second points by treating position information pieces of first points as inputs. In this case, an operation model is the linear regression model.

FIG. 7is a diagram for describing an example of a prediction algorithm according to the present embodiment.FIG. 7represents time series variations of position information pieces of the iron, the left knee, the left foot, the right foot, the neck, and the waist that are to be obtained in a case where position information pieces of the right knee and the hand are input to the regression model. A horizontal axis of each graph illustrated inFIG. 7indicates a time, and a vertical axis indicates an X-coordinate, a Y-coordinate, or a Z-coordinate. In addition, a broken line in each graph indicates a time series variation of position information at the time of learning, and a solid line indicates a time series variation of position information at the time of prediction.

Note that, in a case where the linear regression model is constructed as a probability model, the information processing apparatus30may predict position information pieces of second points by the Bayes' estimation. The method of linear regression is not limited to the Bayes' estimation, and various kinds of models can be used.

Dimension Compression

For example, the information processing apparatus30(e.g. the second position information calculation section354) may predict position information pieces of second points using a dimension-compressed operation model.

For example, the above-described swing operation of the iron is represented by three pieces of information including an X-coordinate, a Y-coordinate, and a Z-coordinate of each of eight regions including the neck, the waist, the right knee, the right foot, the left knee, the left foot, the hand, and the club head of the iron, that is to say, represented by 24-dimensional information in total. The dimension-compressed operation model represents the 24-dimensional information as information in the dimension having a dimension number smaller than 24. For example, Principal Component Analysis (PCA) can be used for dimension compression.

Because there is a strong high-speed condition for the attitude of a person, it is generally known that a multidimensional space represented by attitude parameters (i.e. position information pieces of the respective joints) having a multi-degree of freedom can be represented in a lower-dimensional space. Treating the attitude of a person in a low-dimensional space can be said to be equal to subconsciously considering a skeleton model.

An example of the dimension-compressed operation model will be described below with reference toFIG. 8.

FIG. 8is a diagram illustrating an example of a dimension-compressed operation model according to the present embodiment.FIG. 8illustrates an example of an operation model obtained by compressing 21-dimensional information related to seven regions other than the club head of the iron, into three dimension including a first principal component (PC1), a second principal component (PC2), and a third principal component (PC3). The club head of the iron is excluded for enhancing prediction accuracy. Each plot in a space of the dimension-compressed operation model corresponds to an attitude of a person at each timing in the golf swing. For example, plots50A to50F respectively correspond to attitudes51A to51F.

Subsequently, a prediction algorithm that uses a dimension-compressed operation model will be described with reference toFIG. 9.

FIG. 9is a diagram for describing an example of a prediction algorithm according to the present embodiment. As illustrated inFIG. 9, the information processing apparatus30calculates position information pieces of first points (the right knee and the hand). Subsequently, the information processing apparatus30searches for a point on a compression space in which the position information pieces of the first points are to be reproduced, by a steepest descent method (reference numeral52), and calculates position information pieces of the first points and second points, from the retrieved one point, by PCA inverse mapping (reference numeral53). The information processing apparatus30repeatedly performs these calculations until a difference between the position information pieces of the first points that are obtainable before and after the search falls below a threshold value (reference numeral54). After that, the information processing apparatus30predicts, by linear regression, position information of the club head of the iron on the basis of the position information pieces of the first points and the second points (reference numeral55).

By performing the above-described prediction processing at each timing of the swing, the information processing apparatus30can visualize the entire operation in which the user swings the iron, as illustrated inFIG. 6.

Note that the information processing apparatus30may use Gaussian Process Latent Variable Models (GPLVM), for example, as an alternative dimension compression method of the PCA. The GPLVM is a nonlinear compression method, and is suitable for predicting a more complicated operation although costs of learning and prediction are higher than those in the PCA.

(7) Information Output

For example, the output information may be an image (a moving image or a still image) in which a virtual object corresponding to the target object performs an operation corresponding to the overall operation of the target object. For example, the output information may be an image indicating a time series variation of position information of each point of the user, as illustrated inFIG. 6. Additionally, the output information may be an image in which an avatar corresponding to the user performs an operation corresponding to an operation of the user. A User Interface (UI) example in this case will be described with reference toFIG. 10.

FIG. 10is a diagram for describing an example of a UI according to the present embodiment. As illustrated inFIG. 10, the user holds a controller60for games that includes sensor devices10A and10B, and wears a virtual reality (VR) device61. By varying a position or an attitude of the controller60, the user manipulates an avatar displayed on the VR device61. Specifically, the entire attitude of the user is calculated on the basis of calculation results of position information pieces of the sensor devices10A and10B, and a calculation result of the overall attitude is reflected on a move of the avatar. For example, as illustrated inFIG. 10, when the user holds up the controller60overhead, and then, swings down the controller60, an image62A in which the avatar holds up a sword overhead, and an image62B in which the avatar swings down the sword are displayed on the VR device61. Here, although the sensor devices10are not arranged on the feet of the user, a move of the feet can be reproduced in the avatar by the above-described prediction processing.

Additionally, the output information may be an instruction related to an attachment position of the sensor device10. For example, the information processing apparatus30may output information instructing an optimum attachment position of the sensor device10, on the basis of an analysis result of PCA. Table 1 described below indicates an example of a contribution ratio and a factor loading of each principal component in a swing operation of an iron.

As illustrated in Table 1 described above, a contribution ratio of a first principal component is 65.8%, a cumulative contribution ratio of the first principal component and a second principal component is 94%, and a cumulative contribution ratio of first to third principal components is 99%. In addition, in Table 1 described above, factors are listed in the order of larger factor loadings of each principal component. When the sensor devices10are attached to regions corresponding to factors having larger sums of factor loadings in the first to third principal components, it becomes possible to predict a swing operation using a smaller number of sensor devices10. For example, according to Table 1 described above, in a case where the number of sensor devices10is two, the sensor devices10are desirably attached to the hand and the right knee. In view of the foregoing, the information processing apparatus30outputs information instructing the user that performs a swing operation, to attach the sensor devices10to the hand and the right knee.

<1.5. Flow of Processing>

Subsequently, an example of a flow of operation visualization processing executed in the information processing apparatus30according to the present embodiment will be described with reference toFIG. 11.FIG. 11is a flowchart illustrating an example of a flow of operation visualization processing executed in the information processing apparatus30according to the present embodiment.

As illustrated inFIG. 11, first of all, the information processing apparatus30acquires an operation model corresponding to a context (step S102). For example, the information processing apparatus30acquires an operation model on the basis of a user manipulation, or acquires an operation model corresponding to an identified partial operation of the operations of a target object. Subsequently, the information processing apparatus30acquires sensor information measured and transmitted by the sensor device10(step S104). Subsequently, the information processing apparatus30calculates position information of a first point on the basis of the sensor information (step S106). Then, the information processing apparatus30calculates position information of a second point on the basis of the position information of the first point and the operation model (step S108). Next, the information processing apparatus30generates output information (step S110). For example, the information processing apparatus30generates an image on the basis of the position information of the first point and the position information of the second point, or generates information instructing an optimum attachment position of the sensor device10, on the basis of an analysis result of PCA. Subsequently, the information processing apparatus30outputs the generated output information (step S112).

The processing ends through the above flow.

2. Second Embodiment

The present embodiment is a mode of visualizing an operation of a virtual object.

A system1according to the present embodiment includes an information processing apparatus30. A configuration example of the information processing apparatus30according to the present embodiment will be described below with reference toFIG. 12.

<2.1. Configuration Example of Information Processing Apparatus>

FIG. 12is a block diagram illustrating an example of a configuration of the information processing apparatus30according to the present embodiment. As illustrated inFIG. 12, the information processing apparatus30according to the present embodiment includes the input section320, the output section330, the storage section340, and a control section350. Note that, because the functions of the input section320, the output section330, and the storage section340are similar to those in the first embodiment, the description here will be omitted.

The control section350corresponds to a CPU, a DSP, or the like, and performs processing for providing various types of functions of the information processing apparatus30. The control section350may be regarded as at least one electrical circuit formed so as to be able to execute functional units disclosed inFIG. 12. As illustrated inFIG. 12, the control section350includes the model acquisition section352, the first position information calculation section353, the second position information calculation section354, a keyframe registration section356, and a complementing section357. Note that the control section350can further include structural elements other than these structural elements. In other words, the control section350can also perform operations other than operations of these structural elements. The operations of these structural elements will be described in detail later.

(1) Target Object

In the present embodiment, a target object is a virtual object. In addition, position information in the present embodiment is a three-dimensional coordinate in a virtual space.

For example, the target object may include one or more virtual moving objects.

For example, the target object may include a virtual moving object having a plurality of joints. Then, an operation of the target object may include time series variations of position information pieces of the plurality of joints of the virtual moving object.

In the present embodiment, a target object to be visualized is assumed to be an avatar of a virtual human.

Hereinafter, a manipulator of the information processing apparatus30will also be referred to as a user.

In the present embodiment, among regions of a virtual object, a region manipulated by the user will also be referred to as a first point. In addition, among regions of a virtual object, a region not manipulated by the user will also be referred to as a second point.

(2) Calculation of Position Information that is Based on User Manipulation

(3) Calculation of Position Information that is Based on Operation Model

The information processing apparatus30(e.g. the second position information calculation section354) calculates a time series variation of position information of another region of the target object that follows an operation pattern indicated by an operation model, and corresponds to a time series variation of position information of one or more manipulation target regions. In other words, the information processing apparatus30calculates a time series variation of position information of the second point that follows an operation pattern indicated by an operation model, and corresponds to a time series variation of position information of the first point. It thereby becomes possible for the information processing apparatus30to calculate an overall operation of the avatar that includes time series variations of the position information of the first point and the position information of the second point.

Note that the technical features related to an operation model, the selection of the operation model, and prediction algorithms are similar to those in the first embodiment except that the target object is a virtual object, and calculation of position information that is based on sensor information becomes calculation of position information that is based on manipulation information.

(4) Information Output

UI Example

Typically, the information processing apparatus30according to the present embodiment is used for production support for 3D animation. An example of a UI for production support for 3D animation will be described with reference toFIG. 13.

FIG. 13is a diagram illustrating an example of a UI according to the present embodiment. As illustrated inFIG. 13, an avatar71is displayed on a production screen70A. On the production screen70A, the user selects and drags a right hand72of the avatar71by a pointer73. As a result, as in a production screen70B, not only the dragged right hand72(i.e. the first point), but also a left hand, both feet, and the like (i.e. second points) move in accordance with the move of the right hand72. In this manner, the user can cause the avatar71to perform a natural operation, only by designating a move of a partial region, without finely designating a whole-body move of the avatar71for each region. Accordingly, load on production of 3D animation can be reduced.

Generation of Animation

The information processing apparatus30generates animation using a keyframe method, for example. The keyframe method is a technology of generating a moving image by arranging keyframes at every several frames, and complementing between the keyframes.

Subsequently, the information processing apparatus30(e.g. the complementing section357) generates animation by complementing between keyframes.

<2.3. Flow of Processing>

Subsequently, an example of a flow of operation visualization processing executed in the information processing apparatus30according to the present embodiment will be described with reference toFIG. 14.FIG. 14is a flowchart illustrating an example of a flow of operation visualization processing executed in the information processing apparatus30according to the present embodiment.

As illustrated inFIG. 14, first of all, the information processing apparatus30acquires an operation model corresponding to a context (step S202). For example, the information processing apparatus30acquires an operation model on the basis of a user manipulation. Subsequently, the information processing apparatus30acquires manipulation information indicating a user manipulation instructing a partial operation of the virtual object (step S204). Next, the information processing apparatus30calculates position information of the first point on the basis of the manipulation information (step S206). Then, the information processing apparatus30calculates position information of the second point on the basis of the position information of the first point and the operation model (step S208). Next, the information processing apparatus30generates animation (step S210). For example, the information processing apparatus30generates animation by registering a group of keyframes, and complementing between the keyframes. Subsequently, the information processing apparatus30outputs the generated animation (step S212).

The processing ends through the above flow.

3. Hardware Configuration Example

Finally, a hardware configuration of an information processing apparatus according to each of the embodiments will be described with reference toFIG. 15.FIG. 15is a block diagram illustrating an example of the hardware configuration of the information processing apparatus according to each of the embodiments. Meanwhile, the information processing apparatus900illustrated inFIG. 15may realize the information processing apparatus30illustrated in each ofFIG. 3orFIG. 12, for example. Information processing by the information processing apparatus30according to each of the embodiments is realized according to cooperation between software and hardware described below.

As illustrated inFIG. 15, the information processing apparatus900includes a central processing unit (CPU)901, a read only memory (ROM)902, a random access memory (RAM)903and a host bus904a. In addition, the information processing apparatus900includes a bridge904, an external bus904b, an interface905, an input device906, an output device907, a storage device908, a drive909, a connection port911and a communication device913. The information processing apparatus900may include a processing circuit such as a DSP or an ASIC instead of the CPU901or along therewith.

The CPU901functions as an arithmetic processing device and a control device and controls the overall operation in the information processing apparatus900according to various programs. Further, the CPU901may be a microprocessor. The ROM902stores programs, operation parameters and the like used by the CPU901. The RAM903temporarily stores programs used in execution of the CPU901, parameters appropriately changed in the execution, and the like. The CPU901may form the control section350illustrated inFIG. 3orFIG. 12, for example.

The CPU901, the ROM902and the RAM903are connected by the host bus904aincluding a CPU bus and the like. The host bus904ais connected with the external bus904bsuch as a peripheral component interconnect/interface (PCI) bus via the bridge904. Further, the host bus904a, the bridge904and the external bus904bare not necessarily separately configured and such functions may be mounted in a single bus.

The input device906is realized by a device through which a user inputs information, such as a mouse, a keyboard, a touch panel, a button, a microphone, a switch, and a lever. In addition, the input device906may be a remote control device using infrared ray or other electric waves or external connection equipment such as a cellular phone or a PDA corresponding to operation of the information processing apparatus900, for example. Furthermore, the input device906may include an input control circuit or the like which generates an input signal on the basis of information input by the user using the aforementioned input means and outputs the input signal to the CPU901, for example. The user of the information processing apparatus900may input various types of data or order a processing operation for the information processing apparatus900by operating the input device906. The input device906may form the input section320illustrated inFIG. 3orFIG. 12, for example.

In addition to the above, the input device906can be formed by a device that detects information related to the user. For example, the input device906can include various sensors such as an image sensor (a camera, for example), a depth sensor (a stereo camera, for example), an acceleration sensor, a gyro sensor, a geomagnetic sensor, an optical sensor, a sound sensor, a distance measurement sensor, and a force sensor. Also, the input device906may acquire information related to the state of the information processing apparatus900itself such as the posture and the moving velocity of the information processing apparatus900and information related to a surrounding environment of the information processing apparatus900such as brightness or noise around the information processing apparatus900. Also, the input device906may include a GNSS module that receives a GNSS signal (a GPS signal from a global positioning system (GPS) satellite, for example) from a global navigation satellite system (GNSS) satellite and measures position information including the latitude, the longitude, and the altitude of the device. In addition, the input device906may detect the position through Wi-Fi (registered trademark), transmission and reception to and from a mobile phone, a PHS, a smartphone, or the like, near-field communication, or the like, in relation to the position information.

The output device907is formed by a device that may visually or aurally notify the user of acquired information. As such devices, there is a display device such as a CRT display device, a liquid crystal display device, a plasma display device, an EL display device, a laser projector, an LED projector or a lamp, a sound output device such as a speaker and a headphone, a printer device and the like. The output device907outputs results acquired through various processes performed by the information processing apparatus900, for example. Specifically, the display device visually displays results acquired through various processes performed by the information processing apparatus900in various forms such as text, images, tables and graphs. On the other hand, the sound output device converts audio signals including reproduced sound data, audio data and the like into analog signals and aurally outputs the analog signals. The aforementioned display device and the aforementioned sound output device may form the output section330illustrated inFIG. 3orFIG. 12, for example.

The storage device908is a device for data storage, formed as an example of a storage section of the information processing apparatus900. For example, the storage device908is realized by a magnetic storage device such as an HDD, a semiconductor storage device, an optical storage device, a magneto-optical storage device or the like. The storage device908may include a storage medium, a recording device for recording data on the storage medium, a reading device for reading data from the storage medium, a deletion device for deleting data recorded on the storage medium and the like. The storage device908stores programs and various types of data executed by the CPU901, various types of data acquired from the outside and the like. The storage device908may form the storage section340illustrated inFIG. 3orFIG. 12, for example.

The drive909is a reader/writer for storage media and is included in or externally attached to the information processing apparatus900. The drive909reads information recorded on a removable storage medium such as a magnetic disc, an optical disc, a magneto-optical disc or a semiconductor memory mounted thereon and outputs the information to the RAM903. In addition, the drive909can write information on the removable storage medium.

The connection port911is an interface connected with external equipment and is a connector to the external equipment through which data may be transmitted through a universal serial bus (USB) and the like, for example.

The communication device913is a communication interface formed by a communication device for connection to a network920or the like, for example. The communication device913is a communication card or the like for a wired or wireless local area network (LAN), long term evolution (LTE), Bluetooth (registered trademark) or wireless USB (WUSB), for example. In addition, the communication device913may be a router for optical communication, a router for asymmetric digital subscriber line (ADSL), various communication modems or the like. For example, the communication device913may transmit/receive signals and the like to/from the Internet and other communication apparatuses according to a predetermined protocol, for example, TCP/IP or the like. The communication device913may form the communication section310illustrated inFIG. 3, for example.

Further, the network920is a wired or wireless transmission path of information transmitted from devices connected to the network920. For example, the network920may include a public circuit network such as the Internet, a telephone circuit network or a satellite communication network, various local area networks (LANs) including Ethernet (registered trademark), a wide area network (WAN) and the like. In addition, the network920may include a dedicated circuit network such as an internet protocol-virtual private network (IP-VPN).

Hereinbefore, an example of a hardware configuration capable of realizing the functions of the information processing apparatus900according to each of the embodiments is shown. The respective components may be implemented using universal members, or may be implemented by hardware specific to the functions of the respective components. Accordingly, according to a technical level at the time when each of the embodiments are executed, it is possible to appropriately change hardware configurations to be used.

In addition, a computer program for realizing each of the functions of the information processing apparatus900according to each of the embodiments as described above may be created, and may be mounted in a PC or the like. Furthermore, a computer-readable recording medium on which such a computer program is stored may be provided. The recording medium is a magnetic disc, an optical disc, a magneto-optical disc, a flash memory, or the like, for example. Further, the computer program may be delivered through a network, for example, without using the recording medium.

An embodiment of the present disclosure has been described in detail above with reference toFIG. 1toFIG. 15. As described above, the information processing apparatus30according to the present embodiment acquires an operation model indicating an operation pattern related to a target object, and calculates an overall operation from an identified partial operation of operations of the target object, with reference to the acquired operation model. In the first embodiment, it thereby becomes possible to predict operations of the overall regions of the target object, by sensor information pieces obtained from the sensor devices10attached to partial regions of the target object. In addition, in the second embodiment, it becomes possible to generate animation in which the overall regions of the target object are operated, by a user manipulation of operating a partial region of the target object. In this manner, the information processing apparatus30can visualize an operation of a target object from a smaller amount of information. Note that operation information may be output by specifying, as a target object, a specific section such as an arm, a foot, an upper body, or a lower body of the user. In other words, in this specification, a “target object” is not limited to a target object that is physically or virtually independent.

For example, the first embodiment and the second embodiment of the present disclosure can be appropriately combined. For example, animation indicating a user operation may be generated by registering keyframes on the basis of operations of the user at respective timings, and complementing between the keyframes.

For example, devices described in this specification may be implemented as independent devices, or a part or all thereof may be implemented as separate devices. For example, in the functional configuration example of the information processing apparatus30that is illustrated inFIG. 3 or 12, the storage section340and/or the control section350may be included in a device such as a server that is connected with the input section320and the output section330via a network or the like. In addition, the sensor device10and the information processing apparatus30may be integrally formed.

Note that it is not necessary for the processing described in this specification with reference to the flowchart and the sequence diagram to be executed in the order shown in the flowchart. Some processing steps may be performed in parallel. Further, some of additional steps can be adopted, or some processing steps can be omitted.

An information processing apparatus including:

an acquisition section configured to acquire an operation model indicating an operation pattern related to a target object;

a calculation section configured to calculate an overall operation including an identified partial operation of operations of the target object, with reference to the operation model acquired by the acquisition section; and

an output control section configured to output output information indicating the overall operation of the target object that has been calculated by the calculation section.

The information processing apparatus according to (1), in which the acquisition section acquires the operation model corresponding to a context of the target object.

The information processing apparatus according to (2), in which the acquisition section acquires the operation model on a basis of a user manipulation.

The information processing apparatus according to (2), in which the acquisition section acquires the operation model corresponding to an identified partial operation of operations of the target object.

The information processing apparatus according to (3) or (4), in which the target object includes a user,

the operation model includes a first operation model and a second operation model, and

the acquisition section recognizes the context on a basis of a partial operation of the user that has been identified by sensor information measured by a sensor device arranged on the user, and automatically acquires one of the first operation model and the second operation model in accordance with the recognized context.

The information processing apparatus according to any one of (1) to (5), in which the output information includes an image in which a virtual object corresponding to the target object performs an operation corresponding to the overall operation of the target object.

The information processing apparatus according to (6), in which the target object includes a user, and the output information is displayed by a virtual reality (VR) device worn by the user.

The information processing apparatus according to any one of (1) to (7), in which the target object is a real object, and

the calculation section identifies, on a basis of sensor information measured by a sensor device arranged on the target object, a time series variation of position information of the sensor device.

The information processing apparatus according to (8), in which the calculation section calculates a time series variation of position information of another region of the target object that follows the operation pattern indicated by the operation model, and corresponds to a time series variation of position information of the one or more sensor devices.

The information processing apparatus according to (8) or (9), in which the sensor device is arranged at a joint corresponding to a context, among a plurality of predefined joints.

The information processing apparatus according to (10), in which the sensor devices are arranged at two locations including a hand and a knee, in a case where the context is a swing operation of a golf club.

The information processing apparatus according to any one of (8) to (11), in which the sensor device includes an inertial sensor.

The information processing apparatus according to any one of (8) to (12), in which the sensor device is removably attached to a sensor attachment tool for fixing the sensor device on the target object.

The information processing apparatus according to (1), in which the target object includes a user and an object manipulated by the user.

The information processing apparatus according to (14), in which the calculation section calculates, on a basis of an identified operation of a user, an operation of an object manipulated by the user, with reference to the operation model.

The information processing apparatus according to (15), in which the calculation section calculates, on a basis of an identified swing operation of a user, an operation of a golf club swung by the user, with reference to the operation model related to a swing operation of a golf club.

The information processing apparatus according to any one of (1) to (7), in which the target object is a virtual object, and

the calculation section identifies a partial operation of the virtual object on a basis of manipulation information of a user with respect to the virtual object.

The information processing apparatus according to any one of (1) to (17), in which the target object includes a moving object having a plurality of joints, and

an operation of the target object includes time series variations of position information of the plurality of joints of the moving object.

An information processing method including:

acquiring an operation model indicating an operation pattern related to a target object;

calculating, by a processor, an overall operation including an identified partial operation of operations of the target object, with reference to the acquired operation model; and

outputting output information indicating the calculated overall operation of the target object.

A storage medium storing a program for causing a computer to function as:

an acquisition section configured to acquire an operation model indicating an operation pattern related to a target object;

a calculation section configured to calculate an overall operation including an identified partial operation of operations of the target object, with reference to the operation model acquired by the acquisition section; and

an output control section configured to output output information indicating the overall operation of the target object that has been calculated by the calculation section.

REFERENCE SIGNS LIST