Information processing device, information processing method, and information processing system

An information processing apparatus includes processing circuitry (similarity calculation section) configured to calculate a similarity between a posture model and posture data, wherein the posture model being indicative of a posture in a time series gesture, and the posture data being indicative of a posture in a time series of a subject in moving image data.

TECHNICAL FIELD

The present technology relates to an information processing device, an information processing method, and an information processing system, and particularly to an information processing device, an information processing method, and an information processing system that enables presentation of the similarity between a gesture that serves as a model and a gesture of a subject as moving image data to a user.

BACKGROUND ART

In recent years, various kinds of technologies for analyzing images have been researched and developed. For example, a technology of analyzing the facial expression of a viewer in an image and generating response information indicating the response of the viewer has been developed (refer particularly to Patent Literature 1).

On the other hand, recently, since the environment has been made in which general users can easily create moving image data, a large quantity of moving image data is being created. Thus, it is required to provide a useful application using such moving image data for the users.

CITATION LIST

Patent Literature

SUMMARY OF INVENTION

Technical Problem

An information processing apparatus includes

processing circuitry configured to calculate a similarity between a posture model and posture data, wherein

the posture model being indicative of a posture in a time series gesture, and

the posture data being indicative of a posture in a time series of a subject in moving image data.

Advantageous Effects of Invention

According to the present technology, it is possible to present the similarity between a gesture that serves as a model and a gesture of a subject as moving image data to a user.

DESCRIPTION OF EMBODIMENTS

<Configuration Example of First Embodiment of Information Processing Device>

FIG. 1is a block diagram illustrating a configuration example of hardware of a first embodiment of an information processing device to which the present technology is applied.

In the information processing device10ofFIG. 1, a CPU (Central Processing Unit)11, a ROM (Read Only Memory)12, and a RAM (Random Access Memory)13are connected to one another via a bus14.

To the bus14, an input and output interface15is further connected. To the input and output interface15, an input unit16, an output unit17, a storage unit18, a communication unit19, and a drive20are connected.

The input unit16includes a keyboard, a mouse, a microphone, and the like. The output unit17includes a display, a speaker, and the like. The storage unit18includes a hard disk, a non-volatile memory, and the like. The communication unit19includes a network interface, and the like. The drive20drives a removable medium21such as a magnetic disk, an optical disc, a magneto-optical disc, or a semiconductor memory.

In the information processing device10configured as above, various processes are performed in such a way that the CPU11loads, for example, a program stored in the storage unit18on the RAM13via the input and output interface15and the bus14and executes the program.

In the information processing device10, for example, a model generation process is performed in which a posture model indicating a posture in a time series in a gesture that serves as a model is generated. In addition, in the information processing device10, a similarity calculation process is performed in which the similarity between the posture model and target posture data that is posture data indicating the posture of a subject in a time series as moving image data to be processes is calculated. In other words, the CPU11functions as a model generation processing unit that performs the model generation process or as a similarity calculation processing unit that performs the similarity calculation process.

A program that the CPU11executes can be provided by being recorded on, for example, the removable medium21as a package medium, or the like. In addition, such a program can be provided through a wired or a wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

In the information processing device10, such a program can be installed in the storage unit18via the input and output interface15by loading the removable medium21on the drive20. Further, the program can be received in the communication unit19via a wired or a wireless transmission medium and then installed in the storage unit18. In addition to that, the program can be installed in advance in the ROM12or the storage unit18.

<Configuration Example of Model Generation Processing Unit>

FIG. 2is a block diagram illustrating a configuration example of a model generation processing unit.

The model generation processing unit40ofFIG. 2is constituted by an acquisition section41, a posture estimation section42, a model generation section43, and a storage control section44.

Note that, hereinbelow, the case in which the model generation processing unit40generates one posture model will be described, however, when a plurality of posture models are generated, the same process as that performed when one posture model is generated will be performed for each gesture that serves as a model.

The acquisition section41of the model generation processing unit40reads a plurality of moving image data pieces of a subject that makes a gesture that serves as a model (hereinafter, referred to as model moving image data) from the storage unit18, or acquires the data pieces from an external server, or the like, via the communication unit19. Note that the gesture that serves as a model may be decided in advance, or may be set by an operation of the input unit16by a user. The acquisition section41supplies a plurality of model moving image data pieces to the posture estimation section42.

The posture estimation section42extracts feature amounts of a posture of the subject from each of the plurality of model moving image data pieces in a frame unit, and generates posture data using the feature amounts.

Specifically, for example, the posture estimation section42extracts the locations of eight joints of the upper half of a body from the model moving image data of a frame unit as feature amounts using a technique of obtaining mapping between SIFT feature amounts and the three-dimensional coordinates of human joints using ridge regression. This technique is disclosed in “A Local Basis Representation for Estimating Human Pose from Cluttered Images” by Ankur Agarwal, Bill Triggs for 2006 ACCV (Asian Conference on Computer Vision), or the like.

Note that the posture estimation section42may set the extracted feature amounts to be those that will be unchanged with the camera angle during photographing, by converting the extracted feature amounts into, for example, a polar coordinate system having the coordinates of the neck as the center. In addition, the posture estimation section42may perform vector normalization for the extracted feature amounts and set the feature amounts to be those that will be unchanged with the size of the subject. Further, the posture estimation section42may convert the extracted feature amounts into, for example, a polar coordinate system having the coordinates of the neck as the center and performing vector normalization for the feature amounts so as to be set as feature amounts that will be unchanged with the camera angle during photographing and the size of the subject. The posture estimation section42sets time-series data of the feature amounts of a frame unit to be posture data.

The posture estimation section42supplies posture data of each of the plurality of model moving image data pieces to the model generation section43.

The model generation section43models the posture in a time series in the gesture that serves as a model based on the HMM (Hidden Markov Model), or the like using the posture data of the plurality of model moving image data pieces supplied from the posture estimation section42. The HMM is a technique of stochastically modeling time-series data as a combination of a plurality of states, and is suitable for modeling time-series data having temporal extension and contraction even if the time-series data is of the same model, just as the posture data. The model generation section43supplies a posture model generated by the modeling to the storage control section44.

The storage control section44supplies the posture model supplied from the model generation section43to the storage unit18ofFIG. 1and causes the storage unit to store the model.

<Description of Generation of Posture Model>

FIG. 3is a diagram illustrating the generation of the posture model.

In the example ofFIG. 3, the gesture that serves as a model is a batting form of a designated player.

As illustrated inFIG. 3, the acquisition section41acquires, as model moving image data, a plurality of moving image data pieces of the designated player batting from, for example, a database or a cloud server (online storage) of a search site on the Internet via the communication unit19.

The posture estimation section42generates posture data from each of the plurality of model moving image data pieces, and the model generation section43performs modeling based on the HMM, or the like using the posture data so as to generate a posture model.

<Description of Process of Model Generation Processing Unit>

FIG. 4is a flowchart describing a model generation process of the model generation processing unit40ofFIG. 2.

In Step S11ofFIG. 4, the acquisition section41of the model generation processing unit40acquires the plurality of model moving image data from the storage unit18or from an external server, or the like via the communication unit19. The acquisition section41supplies the plurality of model moving image data pieces to the posture estimation section42.

In Step S12, the posture estimation section42extracts feature amounts from each of the plurality of model moving image data pieces in units of frames, and then generates posture data using the feature amounts. The posture estimation section42supplies the posture data of each of the plurality of model moving image data pieces to the model veneration section43.

In Step S13, the model generation section43performs modeling of postures in a time series in the gesture that serves as a model based on the HMM, or the like, using the posture data of the plurality of model moving image data pieces supplied from the posture estimation section42. The model generation section43supplies the posture model generated by the modeling to the storage control section44.

In Step S14, the storage control section44supplies the posture model supplied from the model generation section43to the storage unit18and causes the storage unit to store the model, and then the process ends.

<Configuration Example of Similarity Calculation Processing Unit>

FIG. 5is a block diagram illustrating a configuration example of the similarity calculation processing unit.

The similarity calculation processing unit60ofFIG. 5is constituted by an image acquisition section61, a model acquisition section62, a posture estimation section63, a similarity calculation section64, and a display control section65.

The image acquisition section61of the similarity calculation processing unit60reads moving image data from the storage unit18based on a command from the input unit16according to an operation of the input unit16by the user or acquires moving image data from an external device (not shown) such as a camera, a server, or the like, via the communication unit19.

Specifically, the user operates the input unit16so as to designate, as moving image data to be processed, moving image data of a subject that makes a gesture for which the similarity to the gesture that serves as a model is desired to be ascertained. According to the operation, the input unit16supplies a command of acquiring the moving image data to be processed designated by the user to the image acquisition section61. Based on the command, the image acquisition section61acquires the moving image data to be processed from the storage unit18or the external device. The image acquisition section61supplies the acquired moving image data to be processed to the posture estimation section63and the display control section65.

The model acquisition section62reads posture model from the storage unit18based on a command from the input unit16according to an operation of the input unit16by the user. Specifically, the user operates the input unit16so as to designate a posture model of the gesture that serves as a model among posture models stored in the storage unit18. According to the operation, the input unit16supplies a command of reading the posture model designated by the user to the model acquisition section62. Based on the command, the model acquisition section62reads the posture model from the storage unit18. The model acquisition section62supplies the read posture model to the posture estimation section63.

The posture estimation section63extracts feature amounts from the moving image data supplied from the image acquisition section61in units of frames and then generates posture data using the feature amounts, in the same manner as the posture estimation section42ofFIG. 2. The posture estimation section63supplies the generated posture data to the similarity calculation section64as target posture data.

The similarity calculation section64calculates the similarity between the posture model and the target posture data based on the target posture data supplied from the posture estimation section63and the posture model supplied from the model acquisition section62.

Specifically, the similarity calculation section64determines a combination of states of the posture model of which the likelihood of each state is at the maximum for the target posture data using the Viterbi algorithm. The similarity calculation section64calculates the similarity by performing a predetermined arithmetic operation such as an average value arithmetic operation, or the like, for the likelihood of each determined state. The similarity calculation section64supplies the similarity to the display control section65.

The display control section65causes the output unit17ofFIG. 1to display various kinds of screens based on the moving image data to be processed supplied from the image acquisition section61, the similarity supplied from the similarity calculation section64, and the like.

Note that the number of moving image data pieces to be processed acquired by the image acquisition section61may be one or plural. When the number of moving image data pieces to be processed is plural, the similarity is obtained for each of the moving image data pieces to be processed.

FIGS. 6 and 7are diagrams showing examples of screens displayed by the display control section65when the number of moving image data pieces to be processed is one, andFIGS. 8 and 9are diagrams showing examples of screens displayed by the display control section65when the number of moving image data pieces to be processed is plural.

In the examples ofFIGS. 6 to 9, posture models of batting forms of player OX, player OO, player XX, and player XO are stored in the storage unit18.

When the number of moving image data pieces to be processed is one, if the user first designates moving image data to be processed by operating the input unit16, the display control section65causes the output unit17to display a screen containing an image71corresponding to the moving image data and a selection field72for selecting a posture model stored in the storage unit18as illustrated inFIG. 6. At the right end of the selection field72, a display button72A for displaying selection candidates of posture models is provided.

At this moment, first, by operating the display button72A using an operation of the input unit16, the user causes the selection field72to display information of the posture models stored in the storage unit18as information of selection candidates of the posture models as illustrated inFIG. 7. Accordingly, in the selection field72, as information of each posture model of batting forms of player OX, player OO, player XX, and player XO, “the batting form of player OX”, “the batting form of player OO”, “the batting form of player XX”, and “the batting form of player XO” are displayed.

Next, by selecting information of a desired posture model (in the example ofFIG. 7, the batting form of player OX) from information of the selection candidates of the posture models using an operation of the input unit16, the user designates the posture model. Accordingly, as illustrated inFIG. 7, the information of the posture model selected by the user in the selection field72is highlighted. In addition, the similarity between the posture model and target posture data is calculated by the similarity calculation section64, and based on the similarity, the similarity (in the example ofFIG. 7, 80%) is displayed as illustrated inFIG. 7.

Accordingly, the user can easily recognize the similarity between the gesture that serves as a model and the gesture of the subject in the moving image data to be processed. In other words, in the example ofFIGS. 6 and 7, the user can easily recognize the similarity between the batting form of player OX that serves as a model and the batting form of the moving image data to be processed. As a result, the user can recognize the state of improvement in his batting form.

On the other hand, when the number of moving image data pieces to be processed is plural (six in the example ofFIGS. 8 and 9), first, if the user designates a plurality of moving image data pieces to be processed with an operation of the input unit16, the display control section65causes the output unit17to display a screen containing an image group81including a plurality of images corresponding to the moving image data pieces and the selection field72as illustrated inFIG. 8.

At this moment, by operating the display button72A with an operation of the input unit16in the same manner as in the case ofFIG. 6, the user causes the selection field72to display information of selection candidates of posture models as illustrated inFIG. 9.

Next, by selecting information of a desired posture model (in the example ofFIG. 9, the batting form of player OX) from information of the selection candidates using an operation of the input unit16, the user designates the posture model. Accordingly, as illustrated inFIG. 9, the information of the posture model selected by the user in the selection field72is highlighted. In addition, the similarity between the posture model and target posture data is calculated by the similarity calculation section64, and based on the similarity, a screen containing the highest similarity (in the example ofFIG. 9, 80%) and an image82based on the moving image data to be processed corresponding to the similarity is displayed on the output unit17as illustrated inFIG. 9.

Accordingly, the user can easily recognize the gesture of the subject in the moving image data to be processed that is the most similar to the gesture that serves as a model. In other words, in the example ofFIGS. 8 and 9, the user can easily recognize the batting form of the moving image data to be processed that is the most similar to the batting form of player OX that serves as a model. As a result, the user can recognize which batting form will be optimum, or the like.

Note that images that are displayed on the screens ofFIGS. 6 to 9and which correspond to and are based on the moving image data may be still images, moving images, or thumbnail images.

<Description of Process of Similarity Calculation Processing Unit>

FIG. 10is a flowchart describing the similarity calculation process by the similarity calculation processing unit60ofFIG. 5. The similarity calculation process is started when the user designates moving image data to be processed by operating the input unit16.

In Step S30ofFIG. 10, based on a command of acquiring moving image data to be processed designated by the user from the input unit16, the image acquisition section61of the similarity calculation processing unit60acquires moving image data to be processed from the storage unit18or moving image data to be processed from an external device (not shown) such as a camera, a server, or the like, via the communication unit19. The image acquisition section61supplies the acquired moving image data to be processed to the posture estimation section63and the display control section65.

In Step S31, the display control section65causes the output unit17ofFIG. 1to display the screen containing the image71(or the image group81) corresponding to the moving image data and the selection field72based on the moving image data to be processed supplied from the image acquisition section61. At this moment, the user operates the display button72A at the right end of the selection field72by operating the input unit16. Accordingly, the input unit16supplies information indicating the operation of the display button72A to the display control section65.

In Step S32, the display control section65determines whether the display button72A has been operated by the user or not, in other words, whether the information indicating the operation of the display button72A has been supplied from the input unit16or not. When it is determined that the display button72A has not been operated in Step S32, the display control section stands by until the display button72A is operated.

On the other hand, when it is determined that the display button72A has been operated in Step S32, the display control section65causes the information of the posture models stored in the storage unit18to be displayed in the selection field72as information of selection candidates of the posture models in Step S33. At this moment, by selecting information of a desired posture model among information of the selection candidates of the posture models displayed in the selection field72with an operation of the input unit16, the user designates the posture model. Accordingly, the input unit16supplies a command of reading the posture model designated by the user to the model acquisition section62.

In Step S34, the model acquisition section62determines whether the posture model has been designated by the user or not, in other words, a command of reading the posture model designated by the user has been supplied from the input unit16or not. In Step S34, when it is determined that the posture model has not yet been designated by the user, the section stands by until a posture model is designated by the user.

In Step S34, when it is determined that the posture model has been designated by the user, the model acquisition section62reads the posture model designated by the user from the storage unit18in Step S35. The model acquisition section62supplies the read posture model to the similarity calculation section64.

In Step S36, the posture estimation section63extracts feature amounts from moving image data of which posture data has not been generated in units of frames among moving image data supplied from the image acquisition section61in the same manner as the posture estimation section42ofFIG. 2, and generates posture data using the feature amounts. The posture estimation section63supplies the generated posture data to the similarity calculation section64as target posture data.

In Step S37, the similarity calculation section64calculates the similarity between the posture model and the target posture data based on the target posture data supplied from the posture estimation section63and the posture model supplied from the model acquisition section62. The posture estimation section63supplies the similarity to the display control section65.

In Step S38, the posture estimation section63determines whether the similarities of all moving image data pieces to be processed have been calculated or not. When it is determined that the similarities of all moving image data pieces to be processed have not yet been calculated in Step S38, the process returns to Step S36, and the processes from Step S36to S38are repeated until the similarities of all moving image data pieces to be processed are calculated.

In Step S39, the display control section65determines whether the number of moving image data pieces to be processed is one or not. When it is determined that the number of moving image data pieces to be processed is one in Step S39, the display control section65causes the similarity supplied from the posture estimation section63to be displayed on the output unit17ofFIG. 1in Step S40. Accordingly, the screen ofFIG. 7is displayed on the output unit17.

On the other hand, when it is determined that the number of moving image data pieces to be processed is not one in Step S39, in other words, when the number of moving image data pieces to be processed is plural, the process advances to Step S41. In Step S41, the display control section65selects a moving image data piece having the highest similarity supplied from the similarity calculation section64among moving image data pieces to be processed supplied from the image acquisition section61.

In Step S42, the display control section65causes the image corresponding to the moving image data piece selected in Step S41and the similarity corresponding thereto to be displayed on the output unit17ofFIG. 1. Accordingly, the screen ofFIG. 9is displayed on the output unit17.

As above, since the information processing device10calculates the similarity between target posture data and a posture model, and causes a screen to be displayed based on the similarity, it is possible to present the user with the similarity between a gesture that serves as a model and a gesture of a subject in moving image data to be processed.

Accordingly, it is possible to recognize to what degree a physical activity in, for example, various kinds of sports, dances, operations of driving equipment of vehicle, aircrafts, craftworks, and the like is similar to a gesture that serves as a model, and to correct the physical activity.

Note that, in the first embodiment, one posture model is designated by a user, but a plurality of posture models may be designated. In this case, the similarities of each of the posture models are calculated, and the display control section65causes information on the posture model with the highest similarity to be displayed with a screen containing an image corresponding to moving image data to be processed and the similarity.

In addition, in the first embodiment, a posture model is learned using a plurality of model moving image data pieces, but a posture model may be learned using posture data of one model moving image data piece. In this case, the posture data of the model moving image data is vectorized, and then turned into a posture model. Note that, even when a posture model is learned using a plurality of model moving image data pieces, the posture model may be generated by vectorizing posture data of the model moving image data, optimizing (equalization, or the like) vectors obtained from the result, and then obtaining a representative vector.

<Configuration Example of First Embodiment of Information Processing System>

FIG. 11is a block diagram illustrating a configuration example of a first embodiment of an information processing system to which the present technology is applied.

As illustrated inFIG. 11, the information processing system90is constituted by a cloud server91and an information processing device92. In the information processing system90, not the information processing device92but the cloud server91generates a posture model, and transmits the model to the information processing device92.

Specifically, the cloud server91performs a model generation process in the same manner as the information processing device10. In addition, the cloud server91transmits a stored predetermined posture model to the information processing device92according to a request from the information processing device92.

The information processing device92requests to the cloud server91transmission of a posture model designated by the user in the same manner as the information processing device10. The information processing device92receives the posture model transmitted from the cloud server91according to the request. The information processing device92performs a similarity calculation process in the same manner as the information processing device10using the received posture model.

<Configuration Example of Second Embodiment of Information Processing Device>

FIG. 12is a block diagram illustrating a configuration example of hardware of a second embodiment of the information processing device to which the present technology is applied.

In the configuration illustrated inFIG. 12, the same reference numerals are given to the same configuration as that ofFIG. 1. Overlapping description will be appropriately omitted.

The configuration of an information processing device100ofFIG. 12has the difference from that ofFIG. 1in that a CPU101is provided instead of the CPU11. The information processing device100sets a gesture of a subject of moving image data designated by a user as a search key to be a gesture that serves as a model, and sets moving image data to be processed of which the similarity is within a predetermined range to be a search result.

Specifically, the CPU101of the information processing device100performs various kinds of processes by loading a program stored in the storage unit18on the RAM13and executing the program via the input and output interface15and the bus14. For example, the CPU101generates a posture model using posture data of moving image data designated by the user as a search key, and after generating target posture data, and performs a search process in which moving image data to be processed for which the similarity between the posture model and the target posture data is within a predetermined range is set to be a search result. In other words, the CPU101functions as a search processing unit that performs a search process.

<Configuration Example of Search Processing Unit>

FIG. 13is a block diagram illustrating a configuration example of a search processing unit120.

In the configuration illustrated inFIG. 13, the same reference numerals are given to the same configuration as those ofFIGS. 2 and 5. Overlapping description will be appropriately omitted.

The search processing unit120ofFIG. 13is constituted by the posture estimation section42, the image acquisition section61, the posture estimation section63, a search key acquisition section121, a model generation section122, a similarity calculation section123, and a display control section124.

The search key acquisition section121of the search processing unit120reads moving image data from the storage unit18or acquires moving image data from an external device (not shown) such as a camera, a server, or the like, via the communication unit19based on a command from the input unit16according to an operation of the input unit16by the user.

Specifically, the user operates the input unit16so as to designate moving image data of a subject making a gesture that serves as a search key as model moving image data. The input unit16supplies a command of acquiring the model moving image data to the search key acquisition section121according to the operation. The search key acquisition section121acquires the model moving image data from the storage unit18or an external device based on the command. The search key acquisition section121supplies the acquired model moving image data to the posture estimation section42.

Using posture data of the model moving image data supplied from the posture estimation section42, the model generation section122vectorizes the posture data. The model generation section122supplies the vector of the posture data obtained from the result to the similarity calculation section123as a posture model.

The similarity calculation section123vectorizes target posture data supplied from the posture estimation section63. The similarity calculation section123calculates the distance between the vector of the target posture data obtained from the result and the posture model supplied from the model acquisition section62, and then calculates the similarity between the posture model and the target posture data based on the distance. The similarity calculation section123supplies the calculated similarity to the display control section124.

The display control section124causes various kinds of screens to be displayed on the output unit17ofFIG. 12based on moving image data to be processed supplied from the image acquisition section61and the similarity, or the like supplied from the similarity calculation section123. For example, the display control section124causes screens displaying images corresponding to moving image data for which the similarity is within a predetermined range, among the moving image data to be processed as search results to be displayed on the output unit17.

FIGS. 14 to 17are diagrams showing examples of screens displayed by the display control section124ofFIG. 13.

First, if the user designates moving image data to be processed by operating the input unit16, the display control section124causes images142-1to142-3corresponding to the moving image data and an input field141into which files of the moving image data of the gesture that serves as a search key are input to be displayed as illustrated inFIG. 14. At the right end of the input field141, a reference button141A that supports an input of the files of moving image data of the gesture that serves as a search key is provided.

If the user operates the reference button141A by operating the input unit16, the display control section124causes a screen containing a folder selection field161, a file display part162, a file name input field163, an open button164, and a cancel button165to be displayed as illustrated inFIG. 15.

In the folder selection field161, a predetermined file name (in the example ofFIG. 15, “photo”) is first displayed. At the right end of the folder selection field161, a display button161A for displaying the file name of a selection candidate folder is provided.

When the user desires to set moving image data of a file in a folder other than the folder of the folder name displayed in the folder selection field161to be a search key, the folder names of folders in the selection candidates is caused to be displayed in the folder selection field161by operating the display button161A with an operation of the input unit16. Then, the user can select the folder name of the folder including the file of the moving image data of a gesture that serves as a search key from the folder names by operating the input unit16. Accordingly, the folder name selected by the user is displayed in the folder selection field161.

In the file display part162, information indicating moving image data of files included in the folder of the folder name displayed in the folder selection field161is displayed. In the example ofFIG. 15, thumbnail images of moving image data are displayed as the information indicating the moving image data. At this moment, by operating the input unit16, the user selects information of a file of moving image data of the gesture that serves as a search key from the information displayed in the file display part162. Accordingly, the selected information is, for example, highlighted as illustrated inFIG. 15.

In the file name input field163, the file name of the file corresponding to the information selected by the user in the file display part162is displayed.

The open button164is a button to be operated when the file of the file name displayed in the file name input field163is input to the input field141. Thus, when the user operates the open button164by operating the input unit16, the file name displayed in the file name input field163is displayed in the input field141and at the right end of the input field141, a search button181for starting search is displayed instead of the reference button141A as illustrated inFIG. 16.

At this moment, by operating the search button181with an operation of the input unit16, the user designates the moving image data of the file of the file name displayed in the input field141as model moving image data. Accordingly, the similarity between the model moving image data and the moving image data to be processed is calculated, and images corresponding to the moving image data to be processed for which the similarity is within a predetermined range are displayed as search results in order from the highest similarity, as illustrated inFIG. 17. In the example ofFIG. 17, among the images142-1to142-3, the images142-2and142-3are displayed in the order of the image142-3and the image142-2.

As above, by inputting a file of moving image data of a subject making a desired gesture as a file of moving image data of a gesture that serves as a search key, the user can search for moving image data of subjects making gestures similar to the foregoing gesture. In other words, in the example ofFIGS. 14 to 17, since the user inputs a file of moving image data of a subject making a desired choreographed dance as a file of moving image data of a gesture that serves as a search key, it is possible to search for moving image data of subjects performing a dance similar to the foregoing dance.

On the other hand, the cancel button165ofFIG. 15is a button to be operated when an input of a file of moving image data of a gesture that serves as a search key is quit. Thus, if the user operates the cancel button165by operating the input unit16, the screen ofFIG. 14is displayed again.

<Description of Process of Search Processing Unit>

FIG. 18is a flowchart describing a search process by the search processing unit120ofFIG. 13. The search process is started when the user designates moving image data to be processed by operating the input unit16.

In Step S51, the image acquisition section61of the search processing unit120acquires moving image data to be processed from the storage unit18or acquires moving image data from an external device (not shown) such as a camera, a server, or the like, via the communication unit19based on a command of acquiring moving image data to be processed designated by the user from the input unit16. The image acquisition section61supplies the acquired moving image data to be processed to the posture estimation section63and the display control section124.

In Step S52, the display control section124causes the screen ofFIG. 14containing the images142-1to142-3and the input field141corresponding to the moving image data to be processed to be displayed based on the moving image data to be processed supplied from the image acquisition section61.

At this moment, the user operates the reference button141A with an operation of the input unit16so as to cause the screen ofFIG. 15containing the folder selection field161, the file display part162, the file name input field163, the open button164, and the cancel button165to be displayed. Then, the user changes the folder name displayed in the folder selection field161by operating the display button161A as necessary.

In addition, the user selects information on a file of moving image data of a gesture that serves as a search key from information indicating moving image data displayed in the file display part162, by operating the input unit16. The user operates the open button164with an operation of the input unit16and inputs the file of the moving image data of the gesture that serves as a search key into the input field141.

Accordingly, the display control section124causes the file name of the file of the moving image data of the gesture that serves as a search key input by the user to be displayed in the input field141, and causes the search button181to be displayed instead of the reference button141A as illustrated inFIG. 16. At this moment, the user operates the search button181by operating the input unit16. According to the operation, the input unit16supplies a command of acquiring the model moving image data of the file input by the user as model moving image data to the search key acquisition section121.

In Step S53, the search key acquisition section121determines whether the search button181has been operated or not, in other words, whether a command of acquiring the model moving image data from the input unit16has been supplied or not. When it is determined that the search button181has not been operated in Step S53, the search key acquisition section stands by until the search button181is operated.

On the other hand, when it is determined that the search button181has been operated in Step S53, the process advances to Step S54. In Step S54, the search key acquisition section121acquires the moving image data of the file input by the user from the storage unit18as the model moving image data, or acquires the data from an external device (not shown) such as a camera, a server, or the like, via the communication unit19. The search key acquisition section121supplies the acquired model moving image data to the posture estimation section42.

In Step S55, the posture estimation section42extracts feature amounts of the posture of the subject from the moving image data supplied from the search key acquisition section121in units of frames, and generates posture data using the feature amounts. The posture estimation section42supplies the posture data of the model moving image data to the model generation section122.

In Step S56, the model generation section122vectorizes the posture data using the posture data of the model moving image data supplied from the posture estimation section42, and learns the vector of the posture data obtained from the result as a posture model. The model generation section122supplies the posture model to the similarity calculation section123.

In Step S57, the posture estimation section63generates target posture data from moving image data of which posture data has not yet been generated from the moving image data to be processed supplied from the image acquisition section61in the same manner as the posture estimation section42. The posture estimation section63supplies the generated target posture data to the similarity calculation section123.

In Step S58, the similarity calculation section123calculates the similarity between the target posture data and the posture model. Specifically, the similarity calculation section123vectorizes the target posture data supplied from the posture estimation section63, and calculates the distance between the vector of the target posture data obtained from the result and the posture model supplied from the model acquisition section62. Then, the similarity calculation section123calculates the similarity of the posture model and the target posture data based on the distance. The similarity calculation section123supplies the calculated similarity to the display control section124.

In Step S59, the similarity calculation section123determines whether the similarities of all moving image data pieces to be processed have been calculated or not. When it is determined that the similarities of all moving image data pieces to be processed have not yet been calculated in Step S59, the process returns to Step S57, and the processes of Steps S57to S59are performed until the similarities of all moving image data pieces to be processed are calculated.

When it is determined that the similarities of all moving image data pieces to be processed have been calculated in Step S59, the process advances to Step S60. In Step S60, the display control section124causes images corresponding to moving image data for which the similarity is within a predetermined range among the moving image data to be processed to be displayed as search results in order from the highest similarity as illustrated inFIG. 17.

As above, the information processing device100calculates the similarity between the target posture data and the posture model of when the moving image data of the gesture that serves as a search key is set to be the model moving image data, and based on the similarity, causes images corresponding to the moving image data of which the similarity is within a predetermined range to be displayed as search results in order from the highest similarity. Thus, the user can perform a search by setting a desired gesture to be a search key. As a result, it is possible to perform a search by setting a gesture that is difficult to be described in sentences, such as, choreography of a dance, the movement of an actor or an actress in a specific scene of a film, a motion of a child, or the like, as a search key.

Note that, in the third embodiment, one model moving image data piece is designated by the user, but a plurality of model moving image data pieces may be designated. In this case, posture data of the plurality of model moving image data pieces are vectorized, and the vectors of the posture data obtained from the result are optimized so as to generate posture models. Note that, in this case, the posture models may be generated based on the HMM in the same manner as in the first and the second embodiments.

<Configuration Example of Second Embodiment of Information Processing System>

FIG. 19is a block diagram illustrating a configuration example of a second embodiment of the information processing system to which the present technology is applied.

As illustrated inFIG. 19, an information processing system200is constituted by a cloud server201and an information processing device202. In the information processing system200, not the information processing device202but the cloud server201generates target posture data, and transmits the data to the information processing device202.

Specifically, the cloud server201acquires moving image data to be processed in the same manner as the information processing device100according to a request from the information processing device202, and generates target posture data. Then, the cloud server201transmits the target posture data to the information processing device202.

The information processing device202request to the cloud server201the transmission of the target posture data that is posture data of moving image data to be processed designated by the user in the same manner as the information processing device100. The information processing device202receives the target posture data transmitted from the cloud server201according to the request. The information processing device202generates a posture model in the same manner as the information processing device100, calculates the similarity between the received target posture data and the posture model, and displays a screen based on the similarity.

Note that, in the first to the fourth embodiments, the similarity between target posture data and a posture model is calculated, but the similarity between a posture model and a feature amount of a characteristic frame (for example, a frame with a large movement of the joints) among target posture data may be calculated.

In addition, in the present specification, the step of describing a program stored on a program recording medium includes a process performed in a time series according to an order described as well as a process performed in parallel or in an individual manner, though not necessarily performed in a time series.

Furthermore, in the present specification, a system means a set of a plurality of constituent elements (such as devices, and modules (components)), regardless of whether all the constituent components are in the same housing. Therefore, a plurality of devices that are accommodated in separate housings and connected via a network and one device in which a plurality of modules are accommodated in one housing are all systems.

In addition, an embodiment of the present technology is not limited to the above-described embodiments, and can be variously modified within a scope not departing from the gist of the present technology.

In the information processing system90, for example, the cloud server91may calculate similarity. In this case, the information processing device92informs the cloud server91of a posture model designated by a user in the same manner as the information processing device10, and generates target posture data and transmits the data to the cloud server91in the same manner as the information processing device10. The cloud server91calculates the similarity based on the posture model designated by the user among posture models obtained by performing a model generation process in the same manner as the information processing device10and the target posture data transmitted from the information processing device92, and transmits the similarity to the information processing device92. The information processing device92causes a screen to be displayed in the same manner as the information processing device10based on the similarity transmitted from the cloud server91.

In addition, also in the information processing system200, the cloud server201may calculate the similarity. In this case, the information processing device202informs the cloud server201of moving image data to be processed designated by the user in the same manner as the information processing device100, and generates a posture model and transmits the model to the cloud server201in the same manner as the information processing device100. The cloud server201generates target posture data that is posture data of the moving image data to be processed designated by the user in the same manner as the information processing device100. Then, the cloud server201calculates the similarity based on the generated target posture data and the posture model transmitted from the information processing device202, and transmits the similarity to the information processing device202. The information processing device202causes a screen to be displayed in the same manner as the information processing device100based on the similarity transmitted from the cloud server201.

Furthermore, the present technology can be configured as below.

According to an information processing apparatus embodiment, the embodiment includes

processing circuitry configured to calculate a similarity between a posture model and posture data, wherein

the posture model being indicative of a posture in a time series gesture, and

the posture data being indicative of a posture in a time series of a subject in moving image data.

According to one aspect, the embodiment, further includes

a posture estimation section that extracts feature amounts of the posture of the subject from the moving image data and generates the posture data using the feature amounts.

According to another aspect,

the feature amounts are selected to

be unchanged with camera angle during photographing, or

be unchanged with a size of the subject, or

be unchanged with both camera angle during photographing and size of the subject.

According to another aspect, the embodiment further includes

a display controller that causes an image corresponding to the moving image data to be displayed on a display and causes the posture model and other candidate posture models to be displayed.

According to another aspect, the embodiment further includes

a display controller that displays a similarity when a number of moving image data pieces to be processed is one, and displays an image corresponding to selected moving image data and corresponding similarity to be displayed after selecting moving image data with a highest similarity.

According to another aspect, the embodiment further includes

a display controller that causes the similarity to be displayed.

According to another aspect, the embodiment further includes

a display controller that causes a selection field to be displayed that includes a plurality of candidate posture models for user selection.

According to another aspect, the embodiment further includes

a display controller that causes an image data to be processed for determining similarity with respect to the posture model, wherein the image data being at least one of a still image, a moving image and a thumbnail image.

According to another aspect, the embodiment further includes

a display controller that causes the posture data to be displayed as an image group of a plurality of moving image data pieces to be processed for determining similarity with respect to the posture model.

According to another aspect,

the processing circuitry includes a cloud server and an information processing device, the cloud server provides the posture model to the information processing device in response to a request from the information processing device.

According to another aspect,

the information processing device calculates the similarity between the posture model and the posture data.

According to another aspect, the embodiment further includes

a display controller that displays user-selected moving image data in order of highest similarity.

According to another aspect,

the user-selected moving image data is selectable within a predetermined range.

According to another aspect, the embodiment further includes

a search key acquisition section that searches moving image data based on an input search key.

According to another aspect,

the search key being a file of moving image data of a gesture, and the search key acquisition section identifies moving image data for other subjects making a similar gesture.

According to another aspect,

the processing circuitry includes a cloud server and an information processing device,

in response to a request from the information processing device, the cloud server

calculates the similarity based on target posture data and the posture model.

According to another aspect,

the information processing device includes in the request the posture model designated by a user and generates the target posture data.

According to another aspect,

the cloud server provides the similarity to the information processing device as a response to the request.

According to a method embodiment, the method includes

calculating with processing circuitry a similarity between a posture model and posture data, wherein

the posture model being indicative of a posture in a time series gesture, and

the posture data being indicative of a posture in a time series of a subject in moving image data.

According to a non-transitory computer readable medium embodiment, the medium includes instructions stored therein that when executed by processing circuitry implements

an information processing method, the method includes

calculating with processing circuitry a similarity between a posture model and posture data, wherein

the posture model being indicative of a posture in a time series gesture, and

the posture data being indicative of a posture in a time series of a subject in moving image data.

According to one embodiment, an information processing device includes

a similarity calculation unit that calculates the similarity between a posture model indicating a posture in a time series in a gesture that serves as a model and target posture data that is posture data indicating a posture in a time series of a subject in moving image data based on the posture model and the target posture data, and

a display control unit that causes a screen to be displayed on a display unit based on the similarity calculated by the similarity calculation unit.

According to one aspect,

the display control unit causes the screen containing an image corresponding to the similarity and the moving image data to be displayed on the display unit.

According to another aspect,

the similarity calculation unit calculates the similarity for each piece of the moving image data based on the posture model and the target posture data of a plurality of the moving image data pieces, and

the display control unit causes the screen containing an image based on moving image data corresponding to the highest similarity among the similarities of the plurality of moving image data pieces to be displayed on the display unit.

According to another aspect,

the similarity calculation unit calculates the similarity for each piece of the moving image data based on the posture model and the target posture data of a plurality of the moving image data pieces, and

the display control unit causes the screen containing an image corresponding to moving image data of which the similarity is within a predetermined range to be displayed on the display unit.

According to another aspect, the embodiment further includes

a model generation unit that generates the posture model from moving image data of the subject making the gesture that serves as a model, and

in which the similarity calculation unit calculates the similarity based on the posture model generated by the model generation unit and the target posture data.

According to another aspect,

the model generation unit generates the posture model by modeling a posture in a time series in the gesture that serves as a model using the posture data of the plurality of moving image data of the subject making the gesture that serves as a model.

According to another aspect,

the model generation unit generates the posture model by vectorizing posture data of at least one piece of moving image data of the subject making the gesture that serves as a model, and by optimizing the vector of the posture data obtained from the result.

According to another aspect, the embodiment further includes

a posture estimation unit that generates the target posture data from the moving image data, and

in which the similarity calculation unit calculates the similarity based on the posture model and the target posture data generated by the posture estimation unit.

According to a method embodiment, a method of an information processing device includes

a similarity calculation step of calculating the similarity between a posture model indicating a posture in a time series in a gesture that serves as a model and target posture data that is posture data indicating a posture in a time series of a subject in moving image data based on the posture model and the target posture data, and

a display control step of causing a screen to be displayed on a display unit based on the similarity calculated in the process of the similarity calculation step.

According to one embodiment, an information processing system includes

a first information processing device having,

a model generation unit that generates, from moving image data of a subject making a gesture that serves as a model, a posture model indicating a posture in a time series in the gesture,

a transmission unit that transmits the posture model,

a second information processing device having,

a reception unit that receives the posture model transmitted from the transmission unit,

a posture estimation unit that generates posture data indicating a posture in a time series of a subject from moving image data as target posture data,

a similarity calculation unit that calculates the similarity between the posture model and the target posture data based on the posture model received by the reception unit and the target posture data generated by the posture estimation unit, and

a display control unit that causes a screen to be displayed on a display unit based on the similarity calculated by the similarity calculation unit.

According to one embodiment, an information processing system includes

a first information processing device having

a posture estimation unit that generates posture data indicating a posture in a time series of a subject from moving image data as target posture data,

a transmission unit that transmits the target posture data generated by the posture estimation unit,

a second information processing device having

a reception unit that receives the target posture data transmitted from the transmission unit,

a model generation unit that generates, from moving image data of a subject making a gesture that serves as a model, a posture model indicating a posture in a time series in the gesture,

a similarity calculation unit that calculates the similarity between the posture model and the target posture data based on the target posture data received by the reception unit and the posture model generated by the model generation unit, and

a display control unit that causes a screen to be displayed on a display unit based on the similarity calculated by the similarity calculation unit.

According to one embodiment, an information processing system includes

a first information processing device having

a model generation unit that generates, from moving image data of a subject making a gesture that serves as a model, a posture model indicating a posture in a time series in the gesture,

a posture reception unit that receives posture data indicating a posture in a time series of a subject generated from moving image data as target posture data,

a similarity calculation unit that calculates the similarity between the posture model and the target posture data based on the posture model generated by the model generation unit and the target posture data received by the posture reception unit, and

a similarity transmission unit that transmits the similarity calculated by the similarity calculation unit, and

a second information processing device having

a posture estimation unit that generates the target posture data from the moving image data,

a posture transmission unit that transmits the target posture data generated by the posture estimation unit,

a similarity reception unit that receives the similarity transmitted from the similarity transmission unit, and

a display control unit that causes a screen to be displayed on a display unit based on the similarity received by the similarity reception unit.

According to one embodiment, an information processing system includes

a first information processing device having

a posture estimation unit that generates posture data indicating a posture in a time series of a subject from moving image data as target posture data,

a model reception unit that receives a posture model that is generated from moving image data of a subject making a gesture that serves as a model and indicates a posture in a time series in the gesture,

a similarity calculation unit that calculates the similarity between the posture model and the target posture data based on the target posture data generated by the posture estimation unit and the posture model generated by the model reception unit, and

a similarity transmission unit that transmits the similarity calculated by the similarity calculation unit, and

a second information processing device having a model generation unit that generates the posture model from the moving image data of the subject making the gesture that serves as a model,

a model transmission unit that transmits the posture model generated by the model generation unit,

a similarity reception unit that receives the similarity transmitted from the similarity transmission unit, and

a display control unit that causes a screen to be displayed on a display unit based on the similarity received by the similarity reception unit.

REFERENCE SIGNS LIST

10INFORMATION PROCESSING DEVICE

43MODEL GENERATION UNIT

63POSTURE ESTIMATION UNIT

64SIMILARITY CALCULATION UNIT

65DISPLAY CONTROL UNIT

90INFORMATION PROCESSING SYSTEM

92INFORMATION PROCESSING DEVICE

100INFORMATION PROCESSING DEVICE

122MODEL GENERATION UNIT

123SIMILARITY CALCULATION UNIT

124DISPLAY CONTROL UNIT

200INFORMATION PROCESSING SYSTEM

202INFORMATION PROCESSING DEVICE