Patent Description:
Conventionally, there is a technology in which a video image captured a state of an instructor performing exercise such as aerobics, yoga, or dance and a video image captured a state of a user performing exercise are displayed side by side, so that the user can easily learn the exercise of the instructor.

Furthermore, in recent years, athletes exercise wearing devices capable of receiving various types of information externally via a network. For example, Patent Literature <NUM> discloses a technique in which, at a place where one athlete is exercising, virtual objects of other athletes who have exercised in the past at the place are superimposed and displayed on a display unit showing the surroundings. Patent Literatur <NUM> discloses an exercise support system including a plurality of imaging devices, a 3D generation section, an evaluation reference memory, and an evaluation section. The imaging devices are configured to capture videos for movements of a person to be evaluated. The 3D generation section is configured to generate 3D model data of the person to be evaluated based on the video captured by each of the imaging devices. The evaluation reference memory is configured to store reference 3D model data that is 3D model data to become an evaluation reference of an exercise. The evaluation section is configured to evaluate the movements of the person to be evaluated by comparing between the reference 3D model data and the 3D model data generated by the 3D generation section in each body part of the person to be evaluated. Patent Literatur <NUM> discloses a display device including an input unit configured to receive a content selection command; a storage unit configured to store an image of a user; a controller configured to extract skeleton information of the user from the user image, search for data of an action of an actor related to a content selected by the content selection command, and extract skeleton information of the actor from an image of the actor included in the searched action data; and a display unit, wherein the controller is further configured to generate new action data including the actor image replaced with the user image by mapping the user skeleton information on the actor skeleton information, and control the display unit to display the new action data.

By the way, if the video in which the instructor is exercising and the video in which the user is exercising can be displayed in a superimposed manner, the user can learn the movement of the instructor more accurately.

The present disclosure has been made in view of such a situation, and an object of the present disclosure is to provide more effective learning content in learning movement of a body.

According to a first aspect, the present invention provides an information processing apparatus according to claim <NUM>. According to a second aspect, the present invention provides an information processing method according to claim <NUM>. According to a third aspect, the present invention provides a program according to claim <NUM>. Further aspects of the present invention are set forth in the dependent claims, the drawings, and the following description. Further, according to an aspect, the present invention provides an information processing apparatus including an adjustment unit configured to generate an adjusted second virtual object by adjusting, on the basis of feature point information of a first person included in a first virtual object reflecting a body motion of the first person, a second virtual object reflecting a body motion of a second person to be superimposed on the first virtual object.

An information processing method according to the present disclosure is an information processing method including: by an information processing apparatus, generating an adjusted second virtual object by adjusting, on the basis of feature point information of a first person included in a first virtual object reflecting a body motion of the first person, a second virtual object reflecting a body motion of a second person to be superimposed on the first virtual object.

A program according to the present disclosure is a program causing a computer to execute processing of generating an adjusted second virtual object by adjusting, on the basis of feature point information of a first person included in a first virtual object reflecting a body motion of the first person, a second virtual object reflecting a body motion of a second person to be superimposed on the first virtual object.

According to the present disclosure, an adjusted second virtual object is generated by adjusting, on the basis of feature point information of a first person included in a first virtual object reflecting a body motion of the first person, a second virtual object reflecting a body motion of a second person to be superimposed on the first virtual object.

Embodiments for carrying out the present disclosure (hereinafter referred to as an embodiment) are now described. Moreover, the description is given in the following order.

<FIG> is a diagram illustrating an example of an outline of an information processing system to which a technology according to the present disclosure is applied.

In the information processing system of <FIG>, a reference digital twin, which is a virtual object reflecting the body motion, such as aerobics, yoga, and dance, of an instructor TE who is a reference person in a studio SU, is superimposed on a user digital twin, which is a virtual object reflecting the body motion of a user ST in a home HO, and is displayed on a device DE in the home HO.

In general, the digital twin refers to an object or an environment in a real space, information indicating a state of the object or the environment, or the like constructed and represented in real time in a virtual space, or a technology therefor. The digital twin in the present embodiment refers to a virtual object in which a skeleton, a body shape, and movement of a person in a real space are reflected in real time on a virtual space. Specifically, the digital twin is three-dimensionally modeled computer graphics (3DCG) of three-dimensional information of a person displayed on a virtual space. The digital twin is generated on the basis of sensor data acquired by sensing the instructor TE and the user ST by one or a plurality of sensors installed in the studio SU or the home HO. The digital twin may be drawn with the skeleton, body shape, and scale of the corresponding person as they are, or may be drawn with the skeleton, body shape, and scale adjusted for the purpose of protecting the privacy of the person.

Hereinafter, the reference digital twin of the instructor TE is referred to as a teacher digital twin, and the user digital twin of the user ST is referred to as a student digital twin as appropriate.

The user ST can learn the movement of the instructor TE more accurately by moving own body while watching the movement of the teacher digital twin superimposed on the student digital twin.

Furthermore, the instructor TE can give an instruction regarding the movement of the user ST to the user ST by viewing the movement of the student digital twin superimposed on the teacher digital twin in the studio SU.

The studio SU and the home HO may directly exchange (transmit and receive) information by wired communication or wireless communication, or may exchange (transmit and receive) information via a mobile edge computing (MEC) server <NUM> or a cloud server <NUM>. In a case where transmission and reception of information are performed by wireless communication, a communication system such as long term evolution (LTE), Wi-Fi (registered trademark), <NUM>, or <NUM> can be applied to a part or the whole of the wireless communication.

An example of a superimposed video in which the teacher digital twin is superimposed on the student digital twin displayed on the device DE in the home HO will be described with reference to <FIG> and <FIG>.

In the state of the screen #<NUM> of <FIG>, on a student digital twin 30ST that is upright, a lattice-shaped teacher digital twin 30TE, which is also upright, is superimposed. In the drawing, a button <NUM>, which is a graphical user interface (GUI) for starting a lesson by the instructor TE in the studio SU, is displayed on the upper right of the screen #<NUM>.

As illustrated in the state of the screen #<NUM>, if the user ST raises one hand and it is determined that the hand of the corresponding student digital twin 30ST overlaps the area of the button <NUM>, the lesson by the instructor TE is started. Here, determination processing based on the positional relationship between the coordinates of the button <NUM> in the virtual space and the coordinates of the hand of the student digital twin 30ST is performed. Therefore, even if the hand of the student digital twin 30ST overlaps the area of the button <NUM> in front view as in the screen #<NUM>, the lesson is not started in a case where the hand of the student digital twin 30ST is deviated from the area of the button <NUM> in the depth direction.

In the state of the screen #<NUM> of <FIG>, the teacher digital twin 30TE in which the instructor TE bends one knee from the upright state and stands on one leg so as to bend the corresponding knee and stand on one leg is displayed. Furthermore, an attention point (fitting point) indicating a body part to be moved in the exercise is superimposed and displayed on the teacher digital twin 30TE and the student digital twin 30ST. Specifically, the fitting points indicating the positions of the waist, the knee, and the heel in the state of standing on one leg of the teacher digital twin 30TE and the fitting points indicating the positions of the waist, the knee, and the heel in the upright state (before standing on one leg) of the student digital twin 30ST are displayed.

In this way, the movement of the user ST can be guided by displaying the fitting points of the teacher digital twin 30TE and the student digital twin 30ST. Note that, in addition to the fitting points, lines and figures that assist and guide the movement of the user ST may be superimposed and displayed on the teacher digital twin 30TE and the student digital twin 30ST.

In the state of the screen #<NUM>, the user ST stands on one leg in accordance with the movement of the teacher digital twin 30TE, so that the fitting points of the student digital twin 30ST match the fitting points of the teacher digital twin 30TE. At this time, an effect video <NUM> that recommends to maintain the posture is displayed around (in the background of) the student digital twin 30ST. Furthermore, an indicator <NUM> indicating the time during which the user ST (student digital twin 30ST) maintains the posture is displayed on the upper left of the screen #<NUM>.

On the screens #<NUM> and #<NUM>, superimposed videos in a front view (the student digital twin 30ST and the teacher digital twin 30TE) are displayed, but superimposed videos at different viewpoints (angles) can also be displayed. As a result, the user ST can confirm the deviation from the movement of the instructor TE in more detail.

Furthermore, an effect video may be superimposed on a part (portion) where there is a difference in movement between the student digital twin 30ST and the teacher digital twin 30TE such that the part is highlighted. Further, on the contrary, an effect video may be superimposed on a part (portion) where movements are matched between the student digital twin 30ST and the teacher digital twin 30TE such that the part is highlighted.

When the exercise as shown in the screens #<NUM> and #<NUM> is repeated and the lesson ends, a pop-up <NUM> showing the result of the lesson is displayed as shown in the state of the screen #<NUM>. In the pop-up <NUM>, a matching rate of the motion is illustrated as the evaluation result of the exercise of the student digital twin 30ST with respect to the teacher digital twin 30TE. The evaluation result of the exercise is not limited to the matching rate, and the degree of achievement according to the level of the exercise or the like may be scored and indicated.

In this manner, the user ST can learn the exercise of the instructor TE and recognizes the degree of achievement of the user's own exercise while watching the superimposed video.

Here, a use case to which the digital twin as described above can be applied will be described with reference to <FIG> illustrates five use cases UC1 to UC5.

In the use case UC1, a digital twin reflecting the teacher's body motion in real time is applied as the teacher digital twin. Further, a digital twin reflecting the student's body motion in real time is applied as the student digital twin.

The use case UC1 can be applied to, for example, a case in which an instructor who is a teacher handles classes such as aerobics, yoga, and dance in real time from a studio to a student who is a user at home (real-time studio class). Note that, in this use case, the teacher can handle a real-time class not only from the studio but also from home or any other space, and the same applies to the subsequent use cases. The use case UC1 can be implemented by a system configuration including devices on a teacher side and a student side and the MEC server <NUM>.

In the use case UC2, as the teacher digital twin, a digital twin in which the teacher's body motion is reflected in real time or a digital twin in which the teacher's body motion reflected in video content captured in advance (recoded content) is reflected is applied. Further, a digital twin reflecting the student's body motion in real time is applied as the student digital twin.

Similarly to the use case UC1, the use case UC2 can be applied to a real-time class such as aerobics, yoga, and dance. However, in the real-time class of the use case UC2, the teacher can proceed by switching between a case where the teacher performs in real time and a case where the teacher shows the video content (presents a digital twin based on the video content). Furthermore, the use case UC2 can also be applied to, for example, a soccer school in which a professional soccer player teaches a junior-level player how to shoot (kick) or dribble. The use case UC2 can be implemented by a system configuration including devices on a teacher side and a student side, the MEC server <NUM>, and the cloud server <NUM> capable of handling video content.

In the use case UC3, a digital twin reflecting the teacher's body motion in real time is applied as the teacher digital twin. Furthermore, as the student digital twin, a digital twin reflecting the body motion of the student appeared in the video content captured in advance is applied.

Similarly to the use case UC1, the use case UC3 can also be applied to a real-time class such as aerobics, yoga, and dance. However, in the real-time class of the use case UC3, the teacher confirms the movement of the student digital twin based on the video content of the student, so that instruction information such as an instruction and advice for the video content of the student can be added in real time. The use case UC3 can also be applied to, for example, a soccer school in which a professional soccer player teaches a junior-level player how to shoot or dribble. The use case UC3 can be implemented by a system configuration including devices on the teacher side and the student side, the MEC server <NUM>, and the cloud server <NUM> capable of handling video content.

In the use case UC4, as the teacher digital twin, a digital twin in which the teacher's body motion is reflected in real time or a digital twin in which the teacher's body motion reflected in video content captured in advance is reflected is applied. Furthermore, as the student digital twin, a digital twin reflecting the body motion of the student appeared in the video content captured in advance is applied.

Similarly to the use case UC1, the use case UC4 can also be applied to a real-time class such as aerobics, yoga, and dance. However, in the real-time class of the use case UC3, the teacher can proceed by switching a case where instruction information such as an instruction or advice for the video content of the student is added and a case where the video content is shown in real time. The use case UC4 can also be applied to, for example, a soccer school in which a professional soccer player teaches a junior-level player how to shoot or dribble. The use case UC4 can be implemented by a system configuration including devices on the teacher side and the student side, the MEC server <NUM>, and the cloud server <NUM> capable of handling video content.

In the use case UC5, as both the teacher digital twin and the student digital twin, a digital twin reflecting the body motion of the student appeared in the video content captured in advance is applied.

The use case UC5 can be applied to, for example, self-conditioning of golf (confirmation of an action such as a swing performed by oneself). Specifically, the student can confirm the action by oneself by superimposing the digital twin based on the current video content on the digital twin based on own past video content as a model (treating the digital twin based on the past video content as the teacher digital twin). The use case UC5 can also be applied to, for example, self-conditioning of shooting and dribbling for a professional soccer player. The use case UC5 can be implemented by a system configuration including devices on the student side, the MEC server <NUM>, and the cloud server <NUM> capable of handling video content.

Hereinafter, a specific configuration and operation of an information processing system to which the technology according to the present disclosure is applied will be described.

<FIG> is a block diagram illustrating a configuration example of an information processing system to which the technology according to the present disclosure is applied.

The information processing system in <FIG> includes a device <NUM> on the teacher side and a device <NUM> on the student side. In the example of <FIG>, the device <NUM> on the teacher side and the device <NUM> on the student side are configured to directly communicate with each other, but may also communicate via the MEC server <NUM> or the cloud server <NUM>.

The device <NUM> on the teacher side is installed in a space such as a studio or a house where a teacher (an instructor or the like) is located.

On the other hand, the device <NUM> on the student side is installed in a space such as a studio or a house where a student (user) is located.

In a case where the device <NUM> on the teacher side and the device <NUM> on the student side are installed in a wide space such as a studio, for example, they are configured as a relatively large device (or system) such as a device having a booth type housing surrounding the periphery of a person or a device having a whole-body mirror type display surface in which the entire body of the person is reflected. On the other hand, in a case where the device <NUM> on the teacher side and the device <NUM> on the student side are installed in a narrow space such as a home, for example, they are configured as a small-scale device (or system) such as a smartphone including various sensors or a display connectable to the smartphone. Note that the device <NUM> on the teacher side and the device <NUM> on the student side may be configured as devices (or systems) of the same scale.

The device <NUM> on the teacher side includes a display unit <NUM>, an operation unit <NUM>, a storage unit <NUM>, a communication unit <NUM>, a sensor unit <NUM>, and a control unit <NUM>.

The display unit <NUM> includes a liquid crystal display, an organic electro-luminescence (EL) display, or the like, and displays the digital twin and various types of information on the basis of the control of the control unit <NUM>.

The operation unit <NUM> includes a touch panel integrated with a display constituting the display unit <NUM>, a physical button provided on a housing of the device <NUM>, a microphone, and the like. The operation unit <NUM> receives an operation by the teacher and supplies operation information corresponding to the operation to the control unit <NUM>.

The storage unit <NUM> stores programs necessary for operating the device <NUM>, various data set in advance by the teacher and desired to be used in the lesson, and the like.

The communication unit <NUM> includes a network interface and the like, and communicates with the device <NUM> on the student side on the basis of the control of the control unit <NUM>.

The sensor unit <NUM> includes one or a plurality of sensors, and supplies various sensor data acquired by sensing the body motion of the teacher to the control unit <NUM>.

For example, the sensor unit <NUM> includes one or a plurality of time of flight (ToF) sensors and an RGB sensors. The control unit <NUM> generates the teacher digital twin on the basis of the ToF data acquired by the ToF sensor and the RGB data (video data) acquired by the RGB sensor. In a case where the sensor unit <NUM> includes a plurality of ToF sensors and RGB sensors, the control unit <NUM> can also generate the teacher digital twin on the basis of the volumetric capture data generated by the volumetric capture using the acquired sensor data. The sensor unit <NUM> may include various sensors capable of acquiring sensor data other than ToF data and RGB data.

The control unit <NUM> executes various processing on the basis of a program stored in the storage unit <NUM>, operation information from the operation unit <NUM>, and information acquired via the communication unit <NUM>.

The control unit <NUM> includes a digital twin generation unit <NUM> and an instruction information generation unit <NUM>. Each functional unit included in the control unit <NUM> is implemented by executing a program stored in the storage unit <NUM>.

Meanwhile, the device <NUM> on the student side includes a display unit <NUM>, an operation unit <NUM>, a storage unit <NUM>, a communication unit <NUM>, a sensor unit <NUM>, and a control unit <NUM>.

The display unit <NUM> includes a liquid crystal display, an organic EL display, or the like, and displays the digital twin and various types of information on the basis of the control of the control unit <NUM>.

The operation unit <NUM> includes a touch panel integrated with a display constituting the display unit <NUM>, a physical button provided on a housing of the device <NUM>, a microphone, and the like. The operation unit <NUM> receives an operation by the student and supplies operation information corresponding to the operation to the control unit <NUM>.

The storage unit <NUM> stores programs necessary for operating the device <NUM>, various data prepared in advance by the student, and the like.

The communication unit <NUM> includes a network interface and the like, and communicates with the device <NUM> on the teacher side on the basis of the control of the control unit <NUM>.

The sensor unit <NUM> includes a plurality of sensors, and supplies various sensor data acquired by sensing the body motion of the student to the control unit <NUM>.

Specifically, the sensor unit <NUM> includes one or a plurality of ToF sensors and an RGB sensors. The sensor unit <NUM> of the device <NUM> on the student side may be configured similarly to the sensor included in the device <NUM> on the teacher side, or may be calibrated from a sensor with a different number or type from the sensor included in the device <NUM> on the teacher side.

The control unit <NUM> includes a digital twin generation unit <NUM>, a digital twin adjustment unit <NUM>, a superimposed video generation unit <NUM>, an evaluation unit <NUM>, an effect generation unit <NUM>, and a display control unit <NUM>. Each functional unit included in the control unit <NUM> is implemented by executing a program stored in the storage unit <NUM>.

As illustrated in <FIG>, each functional unit included in the control unit <NUM> of the device <NUM> on the teacher side and each functional unit included in the control unit <NUM> of the device <NUM> on the student side execute each processing by transmitting and receiving information to and from each other as indicated by arrows in the figure. In <FIG>, the information corresponding to the dashed arrows is actually transmitted and received via the communication unit <NUM> of the device <NUM> on the teacher side and the communication unit <NUM> of the device <NUM> on the student side.

Hereinafter, details of each functional unit included in the device <NUM> (the control unit <NUM>) on the teacher side and each functional unit included in the device <NUM> (the control unit <NUM>) on the student side will be described.

<FIG> is a diagram for explaining details of the digital twin generation unit <NUM> of the device <NUM> on the teacher side and the digital twin generation unit <NUM> of the device <NUM> on the student side.

Note that the digital twin generation unit <NUM> of the device <NUM> on the teacher side and the digital twin generation unit <NUM> of the device <NUM> on the student side are configured in a similar manner, and thus will be described as a digital twin generation unit N61 as illustrated in <FIG>. Furthermore, the sensor unit <NUM> of the device <NUM> on the teacher side and the sensor unit <NUM> of the device <NUM> on the student side will be similarly described as the sensor unit N50.

The digital twin generation unit N61 generates, on the basis of a body motion of a person, a virtual object that performs a body motion similar to that of the person, that is, a digital twin reflecting the body motion of the person. The digital twin generation unit N61 includes a feature point extraction unit N71, a background processing unit N72, and a 3D model generation unit N73.

On the basis of the sensor data from the sensor unit N50, the feature point extraction unit N71 extracts, as feature point information of the person, skeleton information indicating a skeleton and joint points of the person (teacher or student), three-dimensional contour information indicating a three-dimensional contour of the person, and acceleration information indicating a motion of a body of the person. The feature point information is set as data on a time axis that continuously changes with time.

The skeleton information is extracted, for example, by performing skeleton estimation using machine learning or the like. The skeleton estimation may be performed using only one of the ToF data and the RGB data, or may be performed using both the ToF data and the RGB data.

The three-dimensional contour information is extracted on the basis of, for example, a depth image including ToF data.

The acceleration information is calculated on the basis of, for example, displacements of the skeleton and the joint points indicated by the skeleton information. In a case where the person wears an acceleration sensor as one of the sensor units N50 on each part of the body, the acceleration information may be acquired on the basis of the sensor data from the acceleration sensor. The acceleration information also includes left and right information indicating which any of the body parts (hands, arms, legs, etc.) on the left and right side is moving.

These pieces of feature point information are supplied to the background processing unit N72 together with RGB data (video data).

The background processing unit N72 removes the background of the person in the video data on the basis of the feature point information from the feature point extracting unit N71 and the video data. The video data from which the background has been removed is supplied to the 3D model generation unit N73 together with the feature point information.

The 3D model generation unit N73 generates a digital twin of the person on the basis of the video data from which the background has been removed and the feature point information from the background processing unit N72.

First, the 3D model generation unit N73 models a target person on the basis of the three-dimensional contour information to create a three-dimensional model (3D model). Next, the 3D model generation unit N73 associates the skeleton and the joint points indicated by the skeleton information with the created 3D model. As a result, the body motion of the person can be reflected in the 3D model. Then, the 3D model generation unit N73 synthesizes skin data corresponding to human skin with the 3D model.

As the skin data, skin data having different visual texture is prepared for each purpose of body motion of the person. The purpose of the body motion includes, for example, aerobics, yoga, dance, golf, soccer, and the like, and is selected in advance by a teacher or a student. In addition, the purpose of the body motion is not limited to the sports described above, and may include artistic creation activities such as playing a musical instrument such as a guitar or a piano, and operating a potter's wheel in porcelain.

Then, the 3D model generation unit N73 synthesizes skin data corresponding to the selected purpose of the body motion with respect to the 3D model, thereby generating a digital twin of a type corresponding to the purpose. For example, in a case where soccer is selected as the purpose of the body motion, the digital twin for soccer is generated by synthesizing the skin data for soccer with respect to the 3D model. At this time, for the generated digital twin, meta-information indicating the purpose of the body motion (for example, soccer) may be stored in association with the sensor data.

As described above, the digital twin generation unit N61 extracts the feature point information on the basis of the sensor data, and generates a digital twin as a 3D model on the basis of the extracted feature point information. The feature point information extracted on the basis of the sensor data is added to the generated digital twin and output to the subsequent stage.

<FIG> is a diagram illustrating details of the instruction information generation unit <NUM> of the device <NUM> on the teacher side.

The instruction information generation unit <NUM> generates instruction information indicating an instruction or the like for the student on the basis of the operation information corresponding to the operation of the operation unit <NUM> by the teacher and supplies the instruction information to the display control unit <NUM> of the device <NUM> on the student side.

The operation information here includes, for example, setting information for setting a GUI such as the button <NUM> illustrated in the screens #<NUM> and #<NUM> of <FIG>, and setting information for setting fitting points illustrated in the screen #<NUM> of <FIG>. That is, the teacher can set the GUI and the fitting point displayed on the display unit <NUM> of the device <NUM> on the student side by operating the operation unit <NUM>.

In this case, the instruction information generation unit <NUM> generates display information for displaying a GUI or a fitting point as illustrated in <FIG> as the instruction information on the basis of the operation information (setting information). Such display information may be generated, for example, on the basis of display data stored in the storage unit <NUM> or on the basis of display data acquired via the communication unit <NUM>.

Furthermore, the instruction information generation unit <NUM> may generate the instruction information on the basis of the evaluation value from the evaluation unit <NUM> of the device <NUM> on the student side. The evaluation value indicates, for example, an evaluation result (such as a matching rate of motion) of a lesson illustrated in a pop-up <NUM> on the screen #<NUM> in <FIG>, and a comment corresponding to the evaluation value is automatically generated as the instruction information. This comment may be prepared in advance for each evaluation value, and a comment corresponding to the evaluation value may be selected. The comment generated as the instruction information may be integrated with the comment input by the teacher as the operation information corresponding to the operation of the operation unit <NUM>. Note that there is a possibility that the teacher cannot input an appropriate comment only with the evaluation result such as the matching rate of the motion. Therefore, the instruction information generation unit <NUM> may generate the instruction information or receive the input of the comment by the teacher on the basis of the superimposed video or the effect video from the device <NUM> on the student side or the single student digital twin or the RGB data (video data) of the student.

These pieces of instruction information are displayed on the display unit <NUM> on the device <NUM> on the student side under the control of the display control unit <NUM>.

<FIG> is a diagram for explaining details of the digital twin adjustment unit <NUM> of the device <NUM> on the student side.

The digital twin adjustment unit <NUM> generates an adjusted teacher digital twin (adjusted reference digital twin) by adjusting the teacher digital twin from the digital twin generation unit <NUM> to be superimposed with the student digital twin from the digital twin generation unit <NUM>. The generated adjusted teacher digital twin is supplied to the superimposed video generation unit <NUM> and the evaluation unit <NUM>.

Here, the teacher digital twin is adjusted on the basis of the student digital twin so that the teacher digital twin is matched to the student digital twin so that the student who is the user can compare the movement of the student and the movement of the teacher who is the instructor and easily copy the teacher's movement.

Specifically, the digital twin adjustment unit <NUM> changes the feature point information of the teacher included in the teacher digital twin so as to be close to the feature point information of the student on the basis of the feature point information of the student included in the student digital twin.

For example, the size (scale) of the teacher digital twin is adjusted by changing the skeleton information of the teacher digital twin in accordance with the skeleton information of the student digital twin. The left and right information of the teacher digital twin is changed in accordance with the left and right information of the student digital twin, so that the dominant arm and the dominant leg of the teacher digital twin are adjusted. The three-dimensional contour information of the teacher digital twin is changed in accordance with the three-dimensional contour information of the student digital twin, whereby the body shape of the teacher digital twin is adjusted.

Then, the digital twin adjustment unit <NUM> creates the 3D model on the basis of the changed feature point information of the teacher, thereby generating the adjusted teacher digital twin including the adjusted feature point information as the adjusted 3D model. The digital twin adjustment unit <NUM> can generate an adjusted teacher digital twin in a similar manner to the digital twin generation unit N61 in <FIG>.

<FIG> is a diagram for explaining details of the superimposed video generation unit <NUM> of the device <NUM> on the student side.

The superimposed video generation unit <NUM> generates a superimposed video obtained by superimposing the student digital twin from the digital twin generation unit <NUM> on the adjusted teacher digital twin from the digital twin adjustment unit <NUM>, and supplies the generated superimposed video to the effect generation unit <NUM> and the display control unit <NUM>.

Specifically, the superimposed video generation unit <NUM> maps the adjusted teacher digital twin and the student digital twin to a predetermined reference position on the virtual space, and generates the superimposed video by synchronizing them at a predetermined reference time.

The superimposed video is displayed on the display unit <NUM> under the control of the display control unit <NUM>.

<FIG> is a diagram for explaining details of the evaluation unit <NUM> of the device <NUM> on the student side.

The evaluation unit <NUM> calculates an evaluation value of the student digital twin (that is, the body motion of the student) by comparing the student digital twin from the digital twin generation unit <NUM> with the adjusted teacher digital twin from the digital twin adjustment unit <NUM>.

For example, the evaluation unit <NUM> obtains a difference in the contour information (deviation in posture) between the student digital twin and the adjusted teacher digital twin as the evaluation value. Further, the evaluation unit <NUM> obtains a difference in the acceleration information (deviation in movement) between the student digital twin and the adjusted teacher digital twin as the evaluation value. Furthermore, the evaluation unit <NUM> obtains a difference in the fitting points (deviation in posture) between the student digital twin and the adjusted teacher digital twin as the evaluation value.

Among the evaluation values calculated in this manner, the 3D model information representing (visualizing) the difference by the 3D model is supplied to the effect generation unit <NUM>. In addition, among the calculated evaluation values, meta information (a deviation amount, a deviation part, or the like) obtained by converting the difference into a numerical value or a text is supplied to the display control unit <NUM> and the instruction information generation unit <NUM> (the device <NUM> on the teacher side).

<FIG> is a diagram for explaining details of the effect generation unit <NUM> of the device <NUM> on the student side.

On the basis of the evaluation value (3D model information) from the evaluation unit <NUM>, the effect generation unit <NUM> generates an effect video for the superimposed video from the superimposed video generation unit <NUM>. The effect video is, for example, a video for highlighting a part (portion) deviated in the 3D model between the student digital twin and the adjusted teacher digital twin with a predetermined color or texture, a predetermined figure or pattern combined with the background of the student digital twin in a case where there is a deviation, and a line or an afterimage indicating a trajectory of the movement of the student digital twin or the adjusted teacher digital twin.

The effect generation unit <NUM> maps the effect video to a predetermined reference position on the virtual space, synchronizes the effect video at a predetermined reference time, superimposes the effect video on the superimposed video, and supplies the superimposed video to the display control unit <NUM>.

In the effect video, similarly to the skin data, effect videos having different visual textures are prepared for each purpose of the body motion of the person. That is, the effect generation unit <NUM> generates an effect video image of a type corresponding to the selected purpose of the body motion. For example, in a case where soccer is selected as the purpose of the body motion, a type of effect video corresponding to soccer is generated, and in a case where aerobics is selected as the purpose of the body motion, a type of effect video corresponding to aerobics is generated.

As described above, the display control unit <NUM> may cause the display unit <NUM> to display only the superimposed video from the superimposed video generation unit <NUM>, or may cause the display unit <NUM> to display the superimposed video on which the effect video is superimposed, from the effect generation unit <NUM>.

Furthermore, in a case where the effect video is displayed on the display unit <NUM>, the display control unit <NUM> can also switch the effect video displayed on the display unit <NUM> to an effect video or the like of another texture, for example, in accordance with an operation of the user (student). In this case, a plurality of types of effect videos having different textures is prepared for the purpose of one body motion.

Next, operations of the device <NUM> on the teacher side and the device <NUM> on the student side included in the above-described information processing system will be described.

<FIG> is a flowchart for explaining an operation of the device <NUM> on the teacher side when the teacher is performing in a real-time class, for example. The processing of <FIG> is executed, for example, in response to an instruction to start a lesson from a student.

In step S11, the digital twin generation unit <NUM> generates a teacher digital twin on the basis of sensor data of the teacher sensed by the sensor unit <NUM>.

In step S12, the control unit <NUM> controls the communication unit <NUM> to transmit the teacher digital twin generated by the digital twin generation unit <NUM> to the device <NUM> on the student side.

<FIG> is a flowchart for explaining an operation of the device <NUM> on the student side when the teacher is performing in a real-time class, for example. The processing of <FIG> is executed in conjunction with the processing of <FIG>.

In step S21, the digital twin generation unit <NUM> generates a student digital twin on the basis of sensor data of the student sensed by the sensor unit <NUM>.

In step S22, the digital twin adjustment unit <NUM> generates an adjusted teacher digital twin by adjusting the teacher digital twin from the device <NUM> on the student side on the basis of the student digital twin generated by the digital twin generation unit <NUM>.

In step S23, the superimposed video generation unit <NUM> generates a superimposed video in which the student digital twin is superimposed on the adjusted teacher digital twin.

In step S24, the evaluation unit <NUM> calculates the evaluation value of the student digital twin by evaluating the student digital twin using the adjusted teacher digital twin.

In step S25, the effect generation unit <NUM> generates an effect video for the superimposed image on the basis of the 3D model information among the evaluation values calculated by the evaluation unit <NUM>.

In step S26, the display control unit <NUM> causes the display unit <NUM> to display the superimposed video generated by the superimposed video generation unit <NUM> and the effect video generated by the effect generation unit <NUM>.

Meanwhile, the meta information among the evaluation values calculated by the evaluation unit <NUM> is also transmitted to the device <NUM> on the teacher side.

<FIG> is a flowchart for explaining an operation of the device <NUM> on the teacher side based on the evaluation value from the device <NUM> on the student side. The processing of <FIG> is executed in parallel with the processing of <FIG>.

In step S31, the instruction information generation unit <NUM> generates the instruction information on the basis of the evaluation value (meta information) from the device <NUM> on the student side. Specifically, the instruction information generation unit <NUM> generates, as the instruction information, display information indicating the deviation amount or the deviation part of the movement of the student with respect to the movement of the teacher. The display information may include a comment automatically generated according to the evaluation value (meta information) or a comment input by the teacher.

In step S32, the control unit <NUM> controls the communication unit <NUM> to transmit the instruction information generated by the instruction information generation unit <NUM> to the device <NUM> on the student side.

In the device <NUM> on the student side, the instruction information from the device <NUM> on the teacher side is displayed on the display unit <NUM> together with the superimposed video and the effect video by the display control unit <NUM>.

According to the above processing, since the teacher digital twin is adjusted according to the student digital twin, the student can easily copy the movement of the teacher by comparing the own movement with the teacher's movement while watching the superimposed video.

Furthermore, since the effect video based on the difference from the movement of the teacher is superimposed and displayed on the superimposed video, the student can easily recognize the deviation between the own movement and the teacher's movement.

Furthermore, since the instruction information indicating the deviation amount and the deviation part of the movement of the student and the comment corresponding to the deviation amount and the deviation part are displayed together with the effect video, the student can understand how the own movement is specifically deviated and how to move.

As described above, it is possible to provide more effective learning content for the student to learn the movement of the body.

Note that, in the above description, only the evaluation value calculated by the evaluation unit <NUM> is transmitted from the device <NUM> on the student side to the device <NUM> on the teacher side. Alternatively, the superimposed video generated by the superimposed video generation unit <NUM> and the effect video generated by the effect generation unit <NUM> may be transmitted from the device <NUM> on the student side to the device <NUM> on the teacher side. In this case, in the device <NUM> on the teacher side, the superimposed video and the effect video are displayed on the display unit <NUM> under the control of the control unit <NUM>.

As a result, the teacher can easily recognize the deviation between the movement of the teacher and the movement of the student, and can present a more appropriate instruction or advice to the student as the instruction information (comment). Note that the comment for the student may be not only presented as character information but also output as voice information.

As described above, in the information processing system to which the technology according to the present disclosure is applied, <NUM> can be applied as a communication method between devices.

The <NUM> has three features of "high speed and large capacity", "low latency", and "multiple simultaneous connection". These functions can be implemented by a technology called network slicing for virtually dividing (slicing) a network. In <NUM>, data can be transmitted in a high-speed large-capacity network slice (hereinafter, simply referred to as a slice. ) or can be transmitted in a low-latency network slice according to the type and application of data.

Hereinafter, an application example of <NUM> network slicing applied to an information processing system to which the technology according to the present disclosure is applied will be described.

<FIG> is a diagram illustrating an example in which <NUM> network slicing is applied to the information processing system described above. In the drawing, bold line arrows indicate transmission paths supported by <NUM>.

In the example of <FIG>, in the teacher digital twin generated by the digital twin generation unit <NUM>, from the teacher side to the student side (the digital twin adjustment unit <NUM>), the feature point information is transmitted via a low latency slice, and the 3D model is transmitted via a large-capacity slice.

Furthermore, from the student side to the teacher side (instruction information generation unit <NUM>), the evaluation value calculated by the evaluation unit <NUM> is transmitted via a low latency slice, and the superimposed video generated by the superimposed video generation unit <NUM> and the effect video generated by the effect generation unit <NUM> are transmitted via a large-capacity slice.

In this case, the instruction information generation unit <NUM> may generate the instruction information for the student on the basis of the superimposed video from the superimposed video generation unit <NUM> or the effect video from the effect generation unit <NUM>. Furthermore, the superimposed video and the effect video supplied to the instruction information generation unit <NUM> may be displayed on the display unit <NUM> under the control of the control unit <NUM>.

As described above, since the feature point information and the evaluation value required for the real-time property are transmitted via the low latency slice, the followability of the digital twin with respect to the body motion of the teacher and the quickness of the feedback regarding the body motion of the student can be secured.

Incidentally, each functional unit included in the control unit <NUM> and each functional unit included in the control unit <NUM> described above may not be implemented on the device <NUM> on the teacher side and the device <NUM> on the student side, respectively.

As illustrated in <FIG>, the digital twin adjustment unit <NUM> may be implemented on the device <NUM> on the teacher side.

In the example of <FIG>, from the teacher side to the student side (the superimposed video generation unit <NUM> and the evaluation unit <NUM>), among the adjusted teacher digital twin generated by the digital twin adjustment unit <NUM>, the adjusted feature point information is transmitted via a low latency slice, and the adjusted 3D model is transmitted via a large-capacity slice.

Furthermore, from the student side to the teacher side (instruction information generation unit <NUM>), the evaluation value calculated by the evaluation unit <NUM> is transmitted via a low latency slice, and the superimposed video generated by the superimposed video generation unit <NUM> and the effect video generated by the effect generation unit <NUM> are transmitted via a large-capacity slice. Note that the feature point information of the student digital twin generated by the digital twin generation unit <NUM> may be transmitted to the teacher side (the digital twin adjustment unit <NUM>) via a low latency slice.

Although examples in which the digital twin adjustment unit <NUM> is implemented on either the device <NUM> on the teacher side or the device <NUM> on the student side have been described above, it may be implemented on both the devices <NUM>, and <NUM>. Furthermore, the function of the device <NUM> on the teacher side and the function of the device <NUM> on the student side may be switched at a predetermined timing.

As illustrated in <FIG>, the digital twin generation unit <NUM> and the instruction information generation unit <NUM> may be implemented on a MEC server 10TE close to the device <NUM> on the teacher side, and the digital twin generation unit <NUM> to the effect generation unit <NUM> may be implemented on a MEC server 10ST close to the device <NUM> on the student side.

In this case, the device <NUM> on the teacher side transmits the sensing data acquired by the sensor unit <NUM> to the MEC server 10TE (the digital twin generation unit <NUM>). Similarly, the device <NUM> on the student side transmits the sensing data acquired by the sensor unit <NUM> to the MEC server 10ST (the MEC server 10ST).

Note that the MEC server 10TE (the digital twin generation unit <NUM>) may generate the teacher digital twin by extracting feature points from the recoded content stored in the cloud server <NUM>. As a result, the use case UC2 and the use case UC4 in <FIG> are implemented.

In the example of <FIG>, in the teacher digital twin generated by the digital twin generation unit <NUM>, from the MEC server 10TE on the teacher side to the MEC server 10ST (the digital twin adjustment unit <NUM>) on the student side, the feature point information is transmitted via a low latency slice, and the 3D model is transmitted via a large-capacity slice.

Furthermore, from the MEC server 10ST on the student side to the MEC server 10TE (the instruction information generation unit <NUM>) on the teacher side, the evaluation value calculated by the evaluation unit <NUM> is transmitted via a low latency slice, and the superimposed video generated by the superimposed video generation unit <NUM> and the effect video generated by the effect generation unit <NUM> are transmitted via a large-capacity slice.

As illustrated in <FIG>, the digital twin generation unit <NUM> and the instruction information generation unit <NUM> may be implemented on the MEC server 10TE close to the device <NUM> on the teacher side, and the digital twin generation unit <NUM>, the digital twin adjustment unit <NUM>, and the evaluation unit <NUM> may be implemented on a MEC server 10ST close to the device <NUM> on the student side.

In the student digital twin generated by the digital twin generation unit <NUM>, from the MEC server 10ST on the student side to the device <NUM> (the superimposed video generation unit <NUM>) on the student side, the feature point information is transmitted via a low latency slice, and the 3D model is transmitted via a large-capacity slice. Similarly, to the device <NUM> (the superimposed video generation unit <NUM>) on the student side, among the adjusted teacher digital twin generated by the digital twin adjustment unit <NUM>, the adjusted feature point information is transmitted via a low latency slice, and the adjusted 3D model is transmitted via a large-capacity slice.

In addition, the evaluation value calculated by the evaluation unit <NUM> is transmitted from the MEC server 10ST on the student side to the MEC server 10TE (the instruction information generation unit <NUM>) on the teacher side via a low latency slice. Furthermore, from the device <NUM> on the student side to the MEC server 10TE (the instruction information generation unit <NUM>) on the teacher side, the superimposed video generated by the superimposed video generation unit <NUM> and the effect video generated by the effect generation unit <NUM> are transmitted via a large-capacity slice.

As illustrated in <FIG>, the digital twin generation unit <NUM> may be realized on the MEC server 10TE close to the device <NUM> on the teacher side, and the digital twin generation unit <NUM> and the digital twin adjustment unit <NUM> may be realized on a MEC server 10ST close to the device <NUM> on the student side.

In the student digital twin generated by the digital twin generation unit <NUM>, from the MEC server 10ST on the student side to the device <NUM> (the superimposed video generation unit <NUM> and the evaluation unit <NUM>) on the student side, the feature point information is transmitted via a low latency slice, and the 3D model is transmitted via a large-capacity slice. Similarly, to the device <NUM> (the superimposed video generation unit <NUM> and the evaluation unit <NUM>) on the student side, among the adjusted teacher digital twin generated by the digital twin adjustment unit <NUM>, the adjusted feature point information is transmitted via a low latency slice, and the adjusted 3D model is transmitted via a large-capacity slice.

Furthermore, from the device <NUM> on the student side to the device <NUM> (instruction information generation unit <NUM>) on the teacher side, the evaluation value calculated by the evaluation unit <NUM> is transmitted via a low latency slice, and the superimposed video generated by the superimposed video generation unit <NUM> and the effect video generated by the effect generation unit <NUM> are transmitted via a large-capacity slice.

As illustrated in <FIG>, the digital twin generation unit <NUM> may be implemented on the MEC server 10TE close to the device <NUM> on the teacher side, and the digital twin generation unit <NUM> to the evaluation unit <NUM> may be implemented on a MEC server 10ST close to the device <NUM> on the student side.

In addition, the evaluation value calculated by the evaluation unit <NUM> is transmitted from the MEC server 10ST on the student side to the device <NUM> (the instruction information generation unit <NUM>) on the teacher side via a low latency slice. Furthermore, from the device <NUM> on the student side to the device <NUM> (the instruction information generation unit <NUM>) on the teacher side, the superimposed video generated by the superimposed video generation unit <NUM> and the effect video generated by the effect generation unit <NUM> are transmitted via a large-capacity slice.

Note that the superimposed video generated by the superimposed video generation unit <NUM> may be transmitted from the MEC server 10ST on the student side to the device <NUM> (the effect generation unit <NUM>) on the student side via a large-capacity slice. Furthermore, the evaluation value (3D model information) calculated by the evaluation unit <NUM> may be transmitted from the MEC server 10ST on the student side to the device <NUM> (the effect generation unit <NUM>) on the student side via a large-capacity slice.

As illustrated in <FIG>, only the digital twin adjustment unit <NUM> may be implemented on the MEC server 10ST close to the device <NUM> on the student side.

In the example of <FIG>, in the teacher digital twin generated by the digital twin generation unit <NUM>, from the device <NUM> on the teacher side to the MEC server 10ST (the digital twin adjustment unit <NUM>) on the student side, the feature point information is transmitted via a low latency slice, and the 3D model is transmitted via a large-capacity slice.

From the MEC server 10ST on the student side, to the device <NUM> (the superimposed video generation unit <NUM> and the evaluation unit <NUM>) on the student side, among the adjusted teacher digital twin generated by the digital twin adjustment unit <NUM>, the adjusted feature point information is transmitted via a low latency slice, and the adjusted 3D model is transmitted via a large-capacity slice.

Note that the feature point information of the student digital twin generated by the digital twin generation unit <NUM> may be transmitted to the MEC server 10ST (the digital twin adjustment unit <NUM>) on the student side via a low latency slice.

In the example of <FIG>, the digital twin adjustment unit <NUM> is implemented on the MEC server 10ST close to the device <NUM> on the student side, but may be implemented on the MEC server 10TE close to the device <NUM> on the teacher side.

The configuration of the information processing system that realizes the real-time class has been mainly described above. On the other hand, if the teacher digital twin generated in advance can be reproduced, the user (student) at home can take a non-real-time lesson at a desired timing instead of a real-time class.

<FIG> is a block diagram illustrating another configuration example of an information processing system to which the technology according to the present disclosure is applied.

The information processing system in <FIG> includes a cloud server <NUM> and a device <NUM> on the student side. The device <NUM> on the student side in <FIG> is configured similarly to the device <NUM> on the student side described above, but only main functional units are illustrated in <FIG>.

The cloud server <NUM> includes a storage device <NUM> and an instruction information generation unit <NUM>.

The storage device <NUM> stores the teacher digital twin generated in advance, and supplies the teacher digital twin to the device <NUM> on the student side in response to a request from the device <NUM> on the student side.

The instruction information generation unit <NUM> basically has a function similar to that of the instruction information generation unit <NUM> described above, but is different from the instruction information generation unit <NUM> in that the instruction information is automatically generated on the basis of artificial intelligence (AI).

<FIG> is a diagram for explaining details of the storage device <NUM>.

As illustrated in <FIG>, the storage device <NUM> includes a communication unit <NUM>, a storage unit <NUM>, and a control unit <NUM>.

The storage unit <NUM> stores programs necessary for operating the storage device <NUM>, various data prepared in advance, and the like.

Specifically, the storage unit <NUM> stores the real-time performance of the person and the teacher digital twin generated on the basis of the recoded content, and the stored teacher digital twin is read in response to a request from the device <NUM> on the student side.

Furthermore, the storage unit <NUM> may store sensor data and feature point information acquired in advance, and the teacher digital twin may be generated on the basis of the sensor data and the feature point information. Furthermore, a predetermined recoded content may be stored in the storage unit <NUM>, and the teacher digital twin may be generated on the basis of the recoded content.

The control unit <NUM> executes various processing on the basis of the program stored in the storage unit <NUM>. For example, in response to a request from the device <NUM> on the student side, the control unit <NUM> supplies the teacher digital twin stored in the storage unit <NUM> to the device <NUM> on the student side, and generates the teacher digital twin on the basis of the sensor data and the feature point information stored in the storage unit <NUM>.

Also in the above configuration, since the teacher digital twin is adjusted according to the student digital twin, the student can easily copy the movement of the teacher by comparing the own movement with the teacher's movement while watching the superimposed video.

Note that, in the storage device <NUM>, the teacher digital twin, the sensor data, and the feature point information stored in the storage device <NUM> may be managed in association with the person who has performed the body motion reflected in the digital twin, the sensor data, and the feature point information. Furthermore, in the storage device <NUM>, for example, feature point information may be extracted from a game video of a professional soccer player, and a teacher digital twin generated on the basis of skeleton estimation using machine learning or the like may be managed in association with the professional soccer player.

For example, a person ID for specifying a certain person, time information indicating the date and time when the digital twin is generated, genre information indicating the purpose and type of the body motion, and the like are associated with the digital twin reflecting the body motion of the certain person.

As a result, the user to be a student can select a desired person or a digital twin of body motion and take a non-real-time lesson.

Furthermore, the digital twin associated with the person ID may be a target of electronic commerce in a marketplace (electronic market). In this case, in the storage device <NUM>, the metadata of the copyright information including the person ID, the sales price, the sales period, and the like of the digital twin is stored as a database and centrally managed.

As a result, it is possible to manage the copyright of the provider of the digital twin, for example, protecting the provider's own movement such as an instructor who has provided the digital twin as a work or entering a license agreement by the provider with a predetermined company or group.

The <NUM> network slicing can also be applied to the information processing system of <FIG>.

<FIG> is a diagram illustrating an example in which <NUM> network slicing is applied to the information processing system of <FIG>. In the drawing, bold line arrows indicate transmission paths supported by <NUM>.

In the example of <FIG>, the digital twin generation unit <NUM> to the effect generation unit <NUM> are implemented on the MEC server 10ST close to the device <NUM> on the student side.

In this case, the device <NUM> on the student side transmits the sensing data acquired by the sensor unit <NUM> to the MEC server 10ST (the digital twin generation unit <NUM>).

In the example of <FIG>, in the teacher digital twin stored in the storage device <NUM>, from the cloud server <NUM> to the MEC server 10ST (the digital twin adjustment unit <NUM>) on the student side, the feature point information is transmitted via a low latency slice, and the 3D model is transmitted via a large-capacity slice.

Furthermore, from the MEC server 10ST on the student side to the cloud server <NUM> (the instruction information generation unit <NUM>), the evaluation value calculated by the evaluation unit <NUM> is transmitted via a low latency slice, and the superimposed video generated by the superimposed video generation unit <NUM> and the effect video generated by the effect generation unit <NUM> are transmitted via a large-capacity slice.

As illustrated in <FIG>, the digital twin generation unit <NUM>, the digital twin adjustment unit <NUM>, and the evaluation unit <NUM> may be implemented on the MEC server 10ST close to the device <NUM> on the student side.

In addition, the evaluation value calculated by the evaluation unit <NUM> is transmitted from the MEC server 10ST on the student side to the cloud server <NUM> (the instruction information generation unit <NUM>) via a low latency slice. Furthermore, from the device <NUM> on the student side to the cloud server <NUM> (the instruction information generation unit <NUM>), the superimposed video generated by the superimposed video generation unit <NUM> and the effect video generated by the effect generation unit <NUM> are transmitted via a large-capacity slice.

As illustrated in <FIG>, the digital twin generation unit <NUM> and the digital twin adjustment unit <NUM> may be implemented on the MEC server 10ST close to the device <NUM> on the student side.

Furthermore, from the device <NUM> on the student side to the cloud server <NUM> (the instruction information generation unit <NUM>), the evaluation value calculated by the evaluation unit <NUM> is transmitted via a low latency slice, and the superimposed video generated by the superimposed video generation unit <NUM> and the effect video generated by the effect generation unit <NUM> are transmitted via a large-capacity slice.

As illustrated in <FIG>, the digital twin generation unit <NUM> to the evaluation unit <NUM> may be implemented on the MEC server 10ST close to the device <NUM> on the student side.

Note that the feature point information of the student digital twin generated by the digital twin generation unit <NUM> may be transmitted to the teacher side (the digital twin adjustment unit <NUM>) via a low latency slice.

As described above, the <NUM> network slicing can also be applied to the information processing system of <FIG>.

Hereinafter, modifications of the above-described embodiment will be described.

In the above description, as the digital twin, the 3D model combined with the skin data is displayed on the device <NUM> or the like on the student side. In addition, as illustrated in <FIG>, the skeleton image based on the skeleton information may be superimposed and displayed on the 3D model as the digital twin.

In the example of <FIG>, a skeleton image <NUM> representing a skeleton and joint points of a person who is a student is superimposed and displayed on an upright student digital twin 30ST (3D model). In the example of <FIG>, the teacher digital twin 30TE illustrated in <FIG> is not superimposed, but the teacher digital twin 30TE may be further superimposed and displayed on the skeleton image <NUM>.

In the above description, information such as a digital twin, instruction information, and an evaluation value is transmitted and received between the device <NUM> on the teacher side and the device <NUM> on the student side. In addition, for example, status information indicating the progress status of the lesson taken by the student and the state of the student performing the body motion in the lesson may be transmitted and received between the device <NUM> on the teacher side and the device <NUM> on the student side.

<FIG> are diagrams illustrating presentation examples of character information indicating the above-described status information in the device <NUM> on the student side.

In the example of <FIG>, status information indicating that a lesson is started by the student performing an operation for starting the lesson is transmitted from the device <NUM> on the student side to the device <NUM> on the teacher side.

For example, in the state of the screen #<NUM> of <FIG>, character information <NUM> indicating the name of a lesson to be started is displayed together with the student digital twin 30ST standing upright. Furthermore, on the upper right of the screen #<NUM>, a GUI button <NUM> for starting a lesson is displayed as in <FIG>.

As illustrated in the state of the screen #<NUM>, if the student raises one hand and it is determined that the hand of the corresponding student digital twin 30ST overlaps the area of the button <NUM>, the lesson by the instructor TE is started. At this time, the character information <NUM> changes to a particle video <NUM>.

Thereafter, status information indicating the start of a lesson is transmitted to the device <NUM> on the teacher side, and the particles constituting the particle video <NUM> move so as to be sucked into the upper side of the screen #<NUM> as illustrated in the state of the screen #<NUM>.

As described above, the character information <NUM> changes to the particle video <NUM> and moves to the upper side of the screen #<NUM>, so that the student as the user can intuitively understand that the status information indicating the start of the lesson has been transmitted to the device <NUM> on the teacher side.

In the example of <FIG>, status information indicating that the student taking a lesson is too close to the display (display unit <NUM>) and is dangerous is transmitted from the device <NUM> on the student side to the device <NUM> on the teacher side.

For example, in the state of the screen #<NUM> in <FIG>, character information <NUM> indicating the distance between the student and the display is displayed together with the student digital twin 30ST. On the screen #<NUM>, the character information <NUM> indicates that the distance between the student and the display is <NUM>.

When the distance between the student and the display falls below a predetermined threshold value (for example, <NUM>), the character information <NUM> changes to the particle video <NUM> as illustrated in the state of the screen #<NUM>.

Thereafter, status information indicating the student is too close to the display is transmitted to the device <NUM> on the teacher side, and the particles constituting the particle video <NUM> move so as to be sucked into the upper side of the screen #<NUM> as illustrated in the state of the screen #<NUM>.

As described above, the character information <NUM> changes to the particle video <NUM> and moves to the upper side of the screen #<NUM>, so that the student as the user can intuitively understand that the status information indicating the student himself/herself is too close to the display has been transmitted to the device <NUM> on the teacher side.

In the example of <FIG>, status information indicating that the physical load of the student taking a lesson is transmitted from the device <NUM> on the student side to the device <NUM> on the teacher side. The status information indicating the physical load of the student is generated on the basis of a vital sign acquired by a vital sensor provided as the sensor unit <NUM>, for example.

For example, in the state of the screen #<NUM> in <FIG>, character information <NUM> indicating the physical load of the student is displayed together with the student digital twin 30ST. On the screen #<NUM>, the character information <NUM> indicates that the physical load state of the student is "HARD".

When the vital sign of the student exceeds a predetermined limit value, the character information <NUM> changes to a particle video <NUM> as illustrated in the state of the screen #<NUM>.

Thereafter, status information indicating the physical load of the student exceeds a predetermined limit value is transmitted to the device <NUM> on the teacher side, and the particles constituting the particle video <NUM> move so as to be sucked into the upper side of the screen #<NUM> as illustrated in the state of the screen #<NUM>.

As described above, the character information <NUM> changes to the particle video <NUM> and moves to the upper side of the screen #<NUM>, so that the student as the user can intuitively understand that the status information indicating the physical load of the student himself/herself exceeds the limitation has been transmitted to the device <NUM> on the teacher side.

In the above-described examples, the character information indicating the progress status of the lesson or the state of the student is changed to the particle video, but a part of the skeleton image <NUM> superimposed and displayed on the student digital twin 30ST may be changed to the particle video.

For example, when the skeleton image <NUM> corresponding to the portion of the student digital twin which has moved differently from the teacher digital twin changes to the particle video, the student can recognize that he/she has made an erroneous movement.

Note that when the character information changes to the particle video, display colors may change, for example, the black character information may change to the red particle video.

The above-described presentation example can also be applied to, for example, a configuration in which a line manager of a factory monitors the state of on-site production line workers individually. In this case, the line manager can collectively grasp the work situation, the physical load, the mental stress, and the like of the on-site worker, and if there is a possibility that the state of the site worker hinders the work, the line manager can immediately notify the management manager of the factory of the possibility.

In the embodiments described above, face authentication may be performed when starting a lesson or when starting work in a factory, for example. As a result, it is possible for a teacher to avoid offering a lesson to a wrong student, and it is possible for a line manager of a factory to easily grasp an attendance state of an on-site worker.

In the embodiment described above, mainly on the basis of the student digital twin reflecting the body motion of the student who is the user (first person), the teacher digital twin is adjusted so that the teacher digital twin reflecting the body motion of the teacher who is the reference person (second person) matches the student digital twin. Conversely, the student digital twin may be adjusted on the basis of the teacher digital twin so as to match the student digital twin with the teacher digital twin, or the person to be reference (reference person) may be switched between the student and the teacher.

The series of processes described above can be executed by hardware, and can also be executed in software. In the case of executing the series of processes by software, a program forming the software is installed on a computer. Herein, the term computer includes a computer built into special-purpose hardware, a computer able to execute various functions by installing various programs thereon, such as a general-purpose personal computer, for example, and the like.

<FIG> is a block diagram illustrating a hardware configuration example of a computer that executes the series of processes described above according to a program.

In the computer, a central processing unit (CPU) <NUM>, read only memory (ROM) <NUM>, and random access memory (RAM) <NUM> are interconnected by a bus <NUM>.

Additionally, an input/output interface <NUM> is connected to the bus <NUM>. An input unit <NUM>, an output unit <NUM>, a storage unit <NUM>, a communication unit <NUM>, and a drive <NUM> are connected to the input/output interface <NUM>.

The input unit <NUM> includes a keyboard, a mouse, a microphone, and the like, for example. The output unit <NUM> includes a display, a speaker, and the like, for example. The storage unit <NUM> includes a hard disk, non-volatile memory, and the like, for example. The communication unit <NUM> includes a network interface, for example. The drive <NUM> drives a removable medium <NUM> such as a magnetic disk, an optical disc, a magneto-optical disc, or semiconductor memory.

In a computer configured as above, the series of processes described above are performed by having the CPU <NUM> load a program stored in the storage unit <NUM> into the RAM <NUM> via the input/output interface <NUM> and the bus <NUM>, and execute the program, for example.

For example, programs to be executed by the computer (CPU <NUM>) can be recorded and provided in the removable medium <NUM>, which is a packaged medium or the like. In addition, the program can be supplied via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcast.

In the computer, by mounting the removable medium <NUM> onto the drive <NUM>, programs can be installed into the storage unit <NUM> via the input/output interface <NUM>. Programs can also be received by the communication unit <NUM> via a wired or wireless transmission medium and installed into the storage unit <NUM>. In addition, programs can be installed in advance into the ROM <NUM> or the storage unit <NUM>.

Note that a program executed by the computer may be a program in which processing is chronologically carried out in a time series in the order described herein or may be a program in which processing is carried out in parallel or at necessary timing, such as when the processing is called.

In the present specification, steps of describing a program recorded in a recording medium include not only processing performed in chronological order according to the described order, but also processing executed in parallel or individually even if the processing is not necessarily performed in chronological order.

Further, in this specification, a system has the meaning of a set of a plurality of structural elements (such as an apparatus or a module (part)), and does not take into account whether or not all the structural elements are in the same casing. Therefore, the system may be either a plurality of apparatuses, stored in separate casings and connected through a network, or a single device including a plurality of modules within a single casing.

Claim 1:
An information processing apparatus comprising:
an adjustment unit (<NUM>) configured to generate an adjusted second virtual object by adjusting, on a basis of feature point information of a first person included in a first virtual object reflecting a body motion of the first person, characterized in that it further comprises a second virtual object reflecting a body motion of a second person to be superimposed on the first virtual object,
wherein the adjustment unit adjusts at least a scale or a size of the second virtual object on the basis of the feature point information,
a generation unit (<NUM>, <NUM>) configured to generate, on a basis of the feature point information of a person, a virtual object reflecting a body motion of the person,
wherein the generation unit (<NUM>, <NUM>) extracts feature point information of the person on a basis of sensor data obtained by sensing the person,
wherein the feature point information comprises acceleration information indicating a motion of the body of the person.