Patent ID: 12223541

DETAILED DESCRIPTION

A virtual try-on system includes one or more hardware processors configured to function as a learning unit, an acquisition unit, a deriving unit, and a generation unit. The learning unit learns, by machine learning using three-dimensional data of a try-on person, a learning model using a teacher try-on person image as input data, and using a body shape parameter indicating a body shape of the try-on person represented by the teacher try-on person image and compositing position information of a clothing image in the teacher try-on person image as output data. The acquisition unit acquires a try-on person image. The deriving unit derives output data of the try-on person represented by the try-on person image using the try-on person image and the learning model. The generation unit generates a virtual try-on image by compositing the try-on person image and the clothing image using the derived output data. The following describes an embodiment of a virtual try-on system, a virtual try-on method, a virtual try-on computer program product, an information processing device, and learning data stored in a computer-readable medium in detail with reference to the attached drawings.

FIG.1is a schematic diagram of a virtual try-on system1according to the present embodiment.

The virtual try-on system1includes an information processing device10, a terminal device26, and an external server28. The information processing device10, the terminal device26, and the external server28are connected via a communication line34. The communication line34is a well-known network. The information processing device10, the terminal device26, and the external server28are connected to the communication line34in a wired or wireless manner.

The present embodiment describes a case in which the virtual try-on system1includes one information processing device10, one external server28, and one terminal device26by way of example. However, the virtual try-on system1may have a configuration including one or a plurality of the information processing devices10, one or a plurality of the external servers28, and one or a plurality of the terminal devices26.

The terminal device26is a device that displays a virtual try-on image. The virtual try-on image is a composite image obtained by compositing a clothing image on a try-on person image50of a try-on person. In other words, the virtual try-on image is an image representing a state in which the try-on person virtually tries clothes on. In the present embodiment, the terminal device26generates and displays the virtual try-on image using a learning model generated by the information processing device10. Details of the learning model will be described later.

The terminal device26is operated by the try-on person who virtually tries clothes on. The terminal device26is, for example, a well-known personal computer, a portable mobile terminal, a smartphone, and the like.

The try-on person is a target who tries clothes on. It is sufficient that the try-on person is a target who tries clothes on, and may be a living thing or a non-living thing. Examples of the living thing include a person. The living thing is not limited to a person, and may be an animal other than a person such as a dog or a cat. Examples of the non-living thing include a mannequin that mimics a shape of a human body or an animal and other objects, but the non-living thing is not limited thereto. The try-on person may be a living thing or a non-living thing in a state of wearing clothes. The present embodiment describes a case in which the try-on person is a person.

In the present embodiment, in a case of collectively describing the try-on person and a person other than the try-on person, the person is simply referred to as a user.

The clothes are items that can be worn by the try-on person. Examples of the clothes include an upper garment, a skirt, trousers, shoes, a hat, an accessory, a swimming suit, and a wig. The clothes are not limited to an upper garment, a skirt, trousers, shoes, a hat, an accessory, a swimming suit, a wig, and the like.

The terminal device26includes a photographing unit26A, an input unit26B, and a display unit26C. The photographing unit26A photographs the try-on person to obtain a try-on person image51of the try-on person. The try-on person image51is an example of the try-on person image50. The try-on person image51represents the try-on person image50photographed by the terminal device26.

The photographing unit26A is a well-known digital camera. Specifically, the try-on person image51(try-on person image50) is try-on person image data. In the present embodiment, for simplifying the description, the try-on person image data is simply referred to as the try-on person image51(try-on person image50).

A data format of the try-on person image50is not limited. For example, the try-on person image50is an image having a pixel value indicating a color, luminance, and the like of the try-on person defined for each pixel. The try-on person image50is not limited to a bit map image.

The display unit26C displays various images. In the present embodiment, the display unit26C displays a virtual try-on image. The display unit26C is a well-known liquid crystal display (LCD), an organic electro-luminescence (EL), and the like.

The input unit26B receives an operation instruction from the user. The input unit26B is a keyboard, a mouse, an input button, and the like. The display unit26C and the input unit26B may be integrally configured. In this case, the display unit26C and the input unit26B may be configured as a touch panel.

The external server28is a server device that can process a clothing image and big data. In the present embodiment, the external server28stores the clothing image used for the virtual try-on image and a clothing ID for identifying the clothing image that are associated with each other in advance. A plurality of clothes' images (i.e., clothing images) are stored in the external server28in advance in an associated manner with respective clothing IDs. The external server28analyzes purchase information and the like of the user accumulated in various server devices on the Internet by processing big data.

The information processing device10is a device for generating the learning model. The learning model is an application program for deriving, from the try-on person image, a body shape parameter of the try-on person and the compositing position information indicated by the try-on person image. Details of the learning model will be described later. The information processing device10is accessed by one or a plurality of the terminal devices26, and provides the learning model on which learning is performed, output data derived from the learning model, and the like.

The information processing device10includes an information processing unit11and a main body unit16. The information processing unit11includes a control unit12and a storage unit14. The control unit12controls each unit of the device disposed in the information processing device10. The storage unit14is a well-known hard disk device that stores various kinds of data.

The main body unit16includes a display unit18, a photographing unit20, a three-dimensional data acquisition unit21, and an input unit22.

The input unit22receives various operation instructions made by the user. The input unit22is, for example, a keyboard, a touch panel, a pointing device, a mouse, and the like.

The photographing unit20photographs the try-on person to obtain the try-on person image50of the try-on person. In the following description, the try-on person image50photographed by the photographing unit20disposed in the information processing device10is referred to as a teacher try-on person image40.

The teacher try-on person image40is the try-on person image50used for generating the learning model. The photographing unit20is a well-known photographing device that can acquire the teacher try-on person image40. The photographing unit20is, for example, a digital camera.

By way of example,FIG.1illustrates one photographing unit20. However, the information processing device10may have a configuration including a plurality of the photographing units20. In this case, the photographing units20may be disposed at positions at which the photographing units20can photograph the try-on person from photographing directions different from each other. That is, the respective photographing units20may be disposed at positions at which the photographing units20can photograph the teacher try-on person image40at photographing angles different from each other.

The information processing device10may also have a configuration including a plurality of different types of photographing units20. A difference in the type of the photographing unit20means a difference in at least one of an angle of view, a lens (a telephoto lens, a wide-angle lens, a short focus lens, a zoom lens, a fisheye lens, and the like), photographing magnification, resolution, and a manufacturer of the photographing unit20.

The three-dimensional data acquisition unit21is an appliance for obtaining three-dimensional data of the try-on person. The three-dimensional data is, for example, three-dimensional (3D) feature data, surface data, solid data, 3D computer-aided design (CAD) data, a depth map, and the like.

The three-dimensional data acquisition unit21may be a well-known appliance that can obtain the three-dimensional data described above. For example, in a case in which the three-dimensional data is a depth map, the three-dimensional data acquisition unit21is a photographing device that acquires the depth map by photographing. The depth map may be called a distance image in some cases. The depth map is an image in which a distance from the three-dimensional data acquisition unit21is defined for each pixel.

In the present embodiment, the photographing unit20and the three-dimensional data acquisition unit21photograph the try-on person at the same timing. The photographing unit20and the three-dimensional data acquisition unit21are controlled to successively perform photographing at the same time in synchronization with each other by the control unit12. The photographing unit20and the three-dimensional data acquisition unit21then successively output, to the control unit12, the teacher try-on person image40and the three-dimensional data of the try-on person that are obtained by photographing.

In the present embodiment, the photographing unit20and the three-dimensional data acquisition unit21photograph the try-on person at a plurality of photographing angles. Accordingly, the photographing unit20obtains pieces of the teacher try-on person image40corresponding to the respective photographing angles. The three-dimensional data acquisition unit21obtains pieces of the three-dimensional data corresponding to the respective photographing angles.

The photographing angles may be different from each other. For example, each of the photographing angles is a photographing angle for each angle determined in advance at the time when the try-on person is rotated by 360° about a trunk thereof as a rotation axis. Specifically, the photographing angle may be each of the photographing angles from 0° to 360° in increments of a predetermined angle (for example, 5°). The photographing angle of 0° represents, for example, a photographing angle in a state in which the try-on person faces the display unit18from the front. The photographing angle of 180° represents, for example, a photographing angle in a state in which the try-on person turns his/her back to the display unit18.

Specifically, the photographing unit20and the three-dimensional data acquisition unit21are disposed at the respective positions at which the photographing unit20and the three-dimensional data acquisition unit21can photograph the try-on person facing a display surface of the display unit18from a plurality of different photographing directions. By obtaining taken photographed images, the photographing unit20and the three-dimensional data acquisition unit21may obtain the teacher try-on person images40and pieces of the three-dimensional data corresponding to the respective photographing angles.

At the time when the try-on person faces the display surface of the display unit18, the control unit12may perform control to output a message for prompting the try-on person to take a predetermined pose. For example, the control unit12may perform control to output a message for prompting the try-on person to take a pose so as to obtain an image representing a state in which the try-on person is photographed by holding the photographing unit such as a camera by himself/herself to perform photographing (what is called self-photographing).

For example, the control unit12displays the message on the display unit18. Alternatively, for example, the configuration may be such that the main body unit16further includes a speaker, and the control unit12outputs the message from the speaker.

In this case, the photographing unit20can photograph the try-on person taking a pose similar to that at the time of self-photographing, and obtain the teacher try-on person image40of the try-on person taking the pose.

By successively perform photographing, the photographing unit20and the three-dimensional data acquisition unit21can obtain a plurality of the teacher try-on person images40and pieces of the three-dimensional data the photographing times of which are different from each other.

The display unit18is a device that displays various images. The display unit18is, for example, a well-known display device such as a liquid crystal display device and organic electro luminescence (EL). In the present embodiment, the display unit18displays the teacher try-on person image40photographed by the photographing unit20. Accordingly, the try-on person facing the display unit18can visually recognize the display unit18while having a sense of visually recognizing a mirror. Alternatively, the display unit18may display a superimposed image (described later) generated by the control unit12(details will be described later).

The display unit18is, for example, incorporated in one of surfaces of a housing having a rectangular shape. The present embodiment describes a case in which the display unit18is configured to have a size equal to or larger than a life size of a person. The size of the display unit18is not limited to this size.

FIG.2is a schematic diagram illustrating a positional relation between the main body unit16and a try-on person P according to the present embodiment.

The control unit12(not illustrated inFIG.2) displays, on the display unit18, the teacher try-on person image40of the try-on person P, a superimposed image W indicating a state in which the try-on person P tries various clothes on, and the like. By way of example,FIG.2illustrates the superimposed image W of the teacher try-on person image40and a clothing image42. For example, the try-on person P such as a person visually recognizes a superimposed image W2presented on the display unit18from a position facing the display surface of the display unit18. A photographing position and a photographing direction of the photographing unit20and the three-dimensional data acquisition unit21are adjusted in advance so as to be able to photograph the try-on person P facing the display surface of the display unit18.

Returning toFIG.1, the description will be continued. Next, the following describes functions of the information processing device10and the terminal device26included in the virtual try-on system1.FIG.3is an example of a functional block diagram of the information processing device10and the terminal device26included in the virtual try-on system1.

First, the following describes the information processing device10. The information processing device10includes the control unit12, the photographing unit20, the three-dimensional data acquisition unit21, the storage unit14, the display unit18, and the input unit22. The photographing unit20, the three-dimensional data acquisition unit21, the storage unit14, the display unit18, and the input unit22are connected to the control unit12to be able to exchange data and signals.

The storage unit14stores various kinds of data. In the present embodiment, the storage unit14stores learning data14A, a clothing DB14B, and a three-dimensional data DB14C. Details of the learning data14A will be described later.

In the clothing DB14B, a clothing ID is associated with a clothes' image (a clothing image). The clothing ID is identification information for identifying a corresponding clothes' image (clothing image). The clothing DB14B may be stored in the external server28. The present embodiment describes, as an example, a case in which the clothing DB14B associating a plurality of clothing IDs with clothing images corresponding to the respective clothing IDs is stored in the storage unit14in advance. The clothing DB14B stored in the information processing device10may be updated as appropriate by the clothing DB14B stored in the external server28.

In the three-dimensional data DB14C, the three-dimensional data of the try-on person is associated with a teacher try-on person image ID. The teacher try-on person image ID is identification information for identifying the teacher try-on person image40. That is, in the present embodiment, the storage unit14associates each teacher try-on person image40with the three-dimensional data of the try-on person represented by the teacher try-on person image40to be stored. The three-dimensional data DB14C is updated by the control unit12.

The control unit12includes an acquisition unit12A, a communication unit12B, a learning unit12C, and a deriving unit12D. The learning unit12C includes a model learning unit12E, a correction unit12F, an extended image generation unit12G, a display control unit12H, a reception unit12I, and a change unit12J.

Some of or all of the acquisition unit12A, the communication unit12B, the learning unit12C, the deriving unit12D, the model learning unit12E, the correction unit12F, the extended image generation unit12G, the display control unit12H, the reception unit12I, and the change unit12J may be implemented by causing a processing device such as a central processing unit (CPU) to execute a computer program, that is, by software, may be implemented by hardware such as an integrated circuit (IC), or may be implemented by using both of software and hardware, for example.

The communication unit12B communicates with the external server28and the terminal device26via the communication line34.

The acquisition unit12A acquires the teacher try-on person image40. In the present embodiment, the acquisition unit12A acquires the teacher try-on person image40from the photographing unit20. As described above, there may be a case in which the photographing unit20obtains, by photographing, a plurality of the teacher try-on person images40that are different in at least one of a photographing time, a photographing angle, a type of the photographing unit20used for photographing, and a pose of the try-on person at the time of photographing for one try-on person. In this case, the acquisition unit12A acquires a plurality of types of teacher try-on person images40for one try-on person. The acquisition unit12A may also acquire the teacher try-on person image40for each of a plurality of the try-on persons. Alternatively, the acquisition unit12A may acquire one teacher try-on person image40including a plurality of the try-on persons.

The acquisition unit12A may acquire the teacher try-on person image40from the storage unit14or the external server28. The present embodiment describes a case in which the acquisition unit12A acquires the teacher try-on person image40from the photographing unit20by way of example.

The acquisition unit12A also acquires the three-dimensional data of the try-on person. In the present embodiment, the acquisition unit12A acquires the three-dimensional data of the try-on person represented by the teacher try-on person image40from the three-dimensional data acquisition unit21.

The acquisition unit12A also acquires characteristic information of the try-on person input by the input unit22.

The characteristic information represents, for example, an appearance characteristic and an inner characteristic of the try-on person. The appearance characteristic represents a characteristic that is estimated or analyzed based on an external appearance of the try-on person. The inner characteristic represents a characteristic that is not estimated or analyzed, or hardly estimated and analyzed based on the external appearance of the try-on person.

Specifically, the characteristic information represents at least one of a color of skin of the try-on person, an exposure degree from clothing, a hairstyle, nationality, a dominant hand, personality, a recommended photographing direction, a color of eyes, a color of hair, a favorite color, a gender, an age, and a recommended pose. The recommended photographing direction represents a photographing direction preferred by the try-on person. The exposure degree from clothing represents a position and a ratio of a region exposed from the clothing (that is, an externally exposed skin region) in a photographed subject image. The recommended pose represents a pose preferred by the try-on person (preferred posture or movement) at the time of photographing.

For example, the try-on person inputs the characteristic information of the try-on person by operating the input unit22when the photographing unit20photographs the teacher try-on person image40. Accordingly, the acquisition unit12A acquires the characteristic information of the try-on person P indicated by the teacher try-on person image40.

The acquisition unit12A also acquires the try-on person image51, which is photographed by the terminal device26, from the terminal device26via the communication unit12B and the communication line34. The acquisition unit12A also acquires display result information indicating a display result of the virtual try-on image displayed by the terminal device26from the terminal device26via the communication unit12B and the communication line34. Details of the display result information will be described later.

The learning unit12C learns the learning model. As described above, the learning model is an application program for deriving, from the try-on person image50, the body shape parameter of the try-on person and the compositing position information indicated by the try-on person image50. In the present embodiment, the learning model is a learning model using the teacher try-on person image40as input data, and using the body shape parameter indicating the body shape of the try-on person represented by the teacher try-on person image40and the compositing position information of the clothing image in the teacher try-on person image40as output data. In other words, the learning model is an application program for modeling an input/output relation between the teacher try-on person image40, and the body shape parameter and the compositing position information to be calculable. The learning model may also be expressed by a numerical expression such as a function.

In the present embodiment, the learning unit12C learns the learning model by machine learning using the three-dimensional data of the try-on person. Specifically, the learning unit12C learns the learning model by machine learning using the learning data.

FIG.4is a schematic diagram illustrating an example of a data configuration of the learning data14A.

A plurality of pieces of teacher data60are registered in the learning data14A. The learning data14A is subjected to processing performed by the learning unit12C such as registration of a new piece of the teacher data60, and change or correction of the registered teacher data60. Accordingly, a plurality of types of teacher data60(for example, teacher data60A to teacher data60E) are registered in the learning data14A.

The teacher data60includes input data80and output data82corresponding to the input data80. The input data80is data to be a factor of prediction or determination of the corresponding output data82. The output data82is data indicating an answer derived from the corresponding input data80.

In the teacher data60, types of data included in the teacher data60and the output data82are different depending on the type of the teacher data60.

First, the following describes the teacher data60A. The input data80of the teacher data60A includes the teacher try-on person image40. One piece of the teacher data60A includes one piece of the input data80, that is, one piece of the teacher try-on person image40.

The input data80may include characteristic information70. The characteristic information70is information indicating a characteristic of the try-on person. The characteristic information70included in the input data80is information indicating a characteristic of the try-on person represented by the teacher try-on person image40included in the input data80.

The output data82includes a body shape parameter72and compositing position information56.

The body shape parameter72is a parameter indicating the body shape of the try-on person represented by the teacher try-on person image40included in the corresponding input data80.

The body shape parameter includes one or a plurality of parameters. Here, the parameter is a measurement value of one or a plurality of points of a human body. The measurement value is not limited to an actually-measured value, but includes a value obtained by estimating the measurement value and a value corresponding to the measurement value.

Specifically, the body shape parameter includes at least one of parameters of a chest measurement, a waist measurement, a hip measurement, a height, and a shoulder width. The parameters included in the body shape parameter are not limited to these parameters. For example, the body shape parameter may further include parameters such as a length of a sleeve and an inseam.

The body shape parameter may be information indicating the body shape itself of the try-on person, or may be information indicating a size of the clothing assuming that the try-on person wears clothing the size of which matches the body shape. In this case, for example, the body shape parameter may include parameters such as a length of a garment, a shoulder width, a width of a garment, and a length of a sleeve. The present embodiment describes a case of using a length of a garment, a shoulder width, a width of a garment, and a length of a sleeve as the body shape parameter. However, the body shape parameter is not limited thereto.

The compositing position information56is information indicating a compositing position of the clothing image in the teacher try-on person image40included in the corresponding input data80. The compositing position is used for positioning at the time of compositing the clothing image on a region representing the try-on person included in the teacher try-on person image40(hereinafter, referred to as a try-on person region40A).

The compositing position information56may be, for example, information indicating a position of a specific portion of the try-on person (for example, a shoulder portion), or may be information indicating a composite target region of the clothing image in the teacher try-on person image40. The present embodiment describes a case in which the compositing position information represents the composite target region for the clothing image in the teacher try-on person image40.

FIG.5is an explanatory diagram of an example of the compositing position information56. For example, the compositing position information56is information indicating the composite target region having a rectangular shape surrounding the try-on person region40A representing the try-on person in the teacher try-on person image40. The compositing position information56may also be information indicating a position of each apex (56A,56B,56C, and56D) of a rectangle surrounding the try-on person region40A. Alternatively, the compositing position information56may be information indicating a position of at least one of these four apexes.

The compositing position information56is not limited to the information indicating the composite target region having a rectangular shape. For example, the compositing position information56may be information indicating at least part of an external shape of the try-on person region40A included in the teacher try-on person image40.

The compositing position information56may also be information indicating a specific portion of the try-on person (for example, a shoulder portion) in the teacher try-on person image40.

Returning toFIG.4, the description will be continued. The output data82may further include the characteristic information70. That is, at least one of the input data80and the output data82may further include the characteristic information70.

The characteristic information70included in the output data82is characteristic information of the try-on person represented by the teacher try-on person image40included in the input data80corresponding to the output data82.

The teacher data60A to the teacher data60E will be described later. A plurality of types of the teacher data60(the teacher data60A to the teacher data60E) are registered and updated as appropriate through the processing performed by the learning unit12C (details will be described later).

Returning toFIG.3, the description will be continued. The learning unit12C includes the model learning unit12E, the correction unit12F, the extended image generation unit12G, the display control unit12H, the reception unit12I, and the change unit12J.

The model learning unit12E generates and updates the learning data14A, and generates the learning model.

First, the following describes generation of the learning data14A by the model learning unit12E. The model learning unit12E acquires, from the acquisition unit12A, the teacher try-on person image40, the three-dimensional data of the try-on person represented by the teacher try-on person image40, and the characteristic information70of the try-on person. The model learning unit12E then derives the output data82corresponding to the input data80including the teacher try-on person image40and the characteristic information70using the acquired three-dimensional data.

For example, the model learning unit12E creates in advance a function or a model for deriving the body shape parameter indicated by the three-dimensional data from the three-dimensional data. To create the function or the model for deriving the body shape parameter from the three-dimensional data, a well-known method may be used.

By introducing the three-dimensional data acquired by the acquisition unit12A into the function or the model, the model learning unit12E derives the body shape parameter72.

For example, the model learning unit12E derives, as the compositing position information56, a region having a rectangular shape surrounding the try-on person region40A representing the try-on person in the teacher try-on person image40(also refer toFIG.4). The model learning unit12E also obtains the characteristic information70of the try-on person acquired by the acquisition unit12A.

The model learning unit12E then generates, as the output data82, the body shape parameter72, the compositing position information56, and the characteristic information70that are derived.

Furthermore, the model learning unit12E registers, in the learning data14A, the teacher data60A including the input data80including the teacher try-on person image40and the characteristic information70acquired from the acquisition unit12A, and the output data82generated by using the input data80and the three-dimensional data (the body shape parameter72, the compositing position information56, and the characteristic information70).

The model learning unit12E generates the teacher data60A using the method described above every time the acquisition unit12A acquires the teacher try-on person image40, the three-dimensional data, and the characteristic information70. In other words, the model learning unit12E generates the teacher data60A every time the photographing unit20and the three-dimensional data acquisition unit21acquire the teacher try-on person image40and the three-dimensional data by photographing. The model learning unit12E then registers the generated teacher data60A in the learning data14A.

Accordingly, the teacher data60A is generated for each teacher try-on person image40, and registered in the learning data14A of the storage unit14.

Next, the following describes generation of the learning model. The model learning unit12E learns the learning model by machine learning using the learning data14A including a plurality of pieces of the teacher data60. A well-known method may be used for machine learning. For example, the model learning unit12E learns the learning model by deep learning using an algorithm of a convolutional neural network (CNN), recurrent neural network (RNN), and the like.

The learning unit12C may correct or change the teacher data60registered in the learning data14A. In this case, the model learning unit12E may learn the learning model by using the learning data14A including the teacher data60before the correction (for example, the teacher data60A) and the teacher data60after the correction or after the change. The learning unit12C may also learn the learning model using the teacher data60after the correction or after the change.

In the present embodiment, the correction unit12F of the learning unit12C corrects the output data82corresponding to the input data80in accordance with the characteristic information70included in the input data80of the teacher data60A.

For example, as the exposure degree from the clothing indicated by the characteristic information70is higher, the correction unit12F so corrects the body shape parameter72in the output data82corresponding to the input data80including the characteristic information70as to indicate a smaller-sized body shape. At this point, the correction unit12F may correct the body shape parameter72such that a portion of a human body having a higher exposure degree from the clothing (for example, a back and the like) indicated by the characteristic information70illustrates a smaller-sized body shape.

For example, the correction unit12F determines a volume of hair based on a hairstyle indicated by the characteristic information70. The correction unit12F then corrects the body shape parameter72in the output data82corresponding to the input data80including the characteristic information70to illustrate a smaller-sized body shape as the volume of hair increases. At this point, the correction unit12F may also correct the body shape parameter72such that a head part of a human body indicated by the characteristic information70illustrates a smaller size.

For example, the correction unit12F may correct the output data82in accordance with the nationality indicated by the characteristic information70. In this case, for example, the correction unit12F so corrects the body shape parameter72in the output data82corresponding to the input data80including the characteristic information70as to illustrate a smaller-sized body shape as the nationality indicated by the characteristic information70is information indicating nationality of people wearing a larger quantity of clothes or wearing a larger number of layers of clothes.

For example, the correction unit12F may correct the output data82in accordance with the gender indicated by the characteristic information70. In this case, for example, in a case in which the gender indicated by the characteristic information70is female, the correction unit12F corrects the body shape parameter72in the output data82corresponding to the input data80including the characteristic information70to illustrate a body shape having a smaller size. For example, in a case in which the gender indicated by the characteristic information70is male, the correction unit12F corrects the body shape parameter72in the output data82corresponding to the input data80including the characteristic information70to illustrate a body shape having a larger size.

In this way, the correction unit12F of the learning unit12C may correct the output data82of the teacher data60A that is created by using the teacher try-on person image40and the three-dimensional data obtained from the photographing unit20and the three-dimensional data acquisition unit21.

Furthermore, the learning unit12C may generate an extended image by extending the teacher try-on person image40included in the teacher data60registered in the learning data14A.

In this case, the extended image generation unit12G of the learning unit12C generates the extended image. The extended image is an image generated by changing at least part of the teacher try-on person image40registered in the learning data14A. Through the processing performed by the extended image generation unit12G, a plurality of extended images similar to the teacher try-on person image40are generated from one teacher try-on person image40.

The extended image is, for example, at least one of a clothing superimposed image, a mask image, a cutout image, a processed image, and a changed image.

The clothing superimposed image is an image obtained by superimposing the clothing image42on the try-on person region40A included in the teacher try-on person image40. For example, the extended image generation unit12G generates the extended image of the teacher try-on person image40by generating the clothing superimposed image by superimposing the clothing image42that is determined in advance on the teacher try-on person image40registered in the learning data14A. To generate the clothing superimposed image, a well-known image composition method may be used.

The mask image is an image obtained by masking a partial region of the teacher try-on person image40. For example, the extended image generation unit12G generates the extended image of the teacher try-on person image40by generating the mask image by changing a color of a partial region of the teacher try-on person image40registered in the learning data14A into a color that is determined in advance such as black or white. To generate the mask image, a well-known image processing method may be used.

The cutout image is an image obtained by cutting out a partial region of the teacher try-on person image40. For example, the extended image generation unit12G generates the extended image of the teacher try-on person image40by generating the cutout image by cutting out a partial region of the teacher try-on person image40registered in the learning data14A. To generate the cutout image, a well-known image processing method may be used.

The processed image is an image obtained by processing at least a partial region of the teacher try-on person image40. To process the image, a well-known processing technique may be used to obtain a style of painting such as a sketch image style, a crayon image style, a painting style such as oil painting and a Japanese painting, or a line drawing/drawing image style. To generate the processed image, a well-known image processing method may be used.

The changed image is an image obtained by changing at least a partial region of the teacher try-on person image40into another image. For example, the changed image is an image obtained by changing a background region as a region other than the try-on person region40A included in the teacher try-on person image40into another image. To generate the changed image, a well-known image processing method may be used.

The extended image generation unit12G then registers the generated extended image in the learning data14A as a new piece of the teacher data60. Specifically, the extended image generation unit12G specifies the teacher data60A including the teacher try-on person image40that is used for generating the extended image. The extended image generation unit12G then uses the characteristic information70included in the specified teacher data60A and the generated extended image as a new piece of the input data80. The extended image generation unit12G uses the output data82included in the specified teacher data60A as the output data82corresponding to the new piece of the input data80. Furthermore, the extended image generation unit12G registers the teacher data60including the new piece of the input data80and the output data82in the learning data14A to update the learning data14A.

Accordingly, as illustrated inFIG.4, the teacher data60C including the extended image52obtained by extending the teacher try-on person image40as the input data80is registered in the learning data14A as a new piece of the teacher data60.

In this case, the model learning unit12E can learn the learning model by using a plurality of types of the teacher data60(including the teacher data60A and the teacher data60C) registered in the learning data14A.

Returning toFIG.3, the description will be continued. The learning unit12C may change the output data82registered in the learning data14A in accordance with a change instruction for the output data82input by an operation instruction on the input unit22made by the user.

In this case, the display control unit12H of the learning unit12C displays, on the display unit18, the superimposed image W obtained by superimposing the clothing image42that is shape-modified in accordance with the body shape parameter72included in the output data82on the compositing position indicated by the compositing position information56included in the output data82corresponding to the input data80including the teacher try-on person image40on the teacher try-on person image40included in the teacher data60A.

The clothing image42that is shape-modified in accordance with the body shape parameter72is obtained by processing of enlarging, reducing, or shape-modifying the clothing image42stored in advance to indicate the size of the body shape indicated by the body shape parameter72. To perform the processing of shape-modifying the clothing image42, a well-known technique may be used.

Thus, for example, the superimposed image W illustrated inFIG.2is displayed on the display unit18. The user refers to the superimposed image W displayed on the display unit18, and inputs a change instruction for at least one of the compositing position information56and the body shape parameter72. The change instruction includes, for example, information after changing at least one of a position, a size, and a range of the compositing position information56, and information after changing the body shape parameter72. The reception unit12I of the learning unit12C then receives the change instruction from the input unit22via the acquisition unit12A.

Returning toFIG.3, the description will be continued. The change unit12J of the learning unit12C changes, into the information after the change indicated by the change instruction received by the reception unit12I, the output data82corresponding to the input data80including the teacher try-on person image40that is used for the last superimposed image W displayed on the display unit18in the teacher data60A.

In this case, the model learning unit12E can learn the learning model by using a plurality of types of the teacher data60(including the teacher data60A, the teacher data60A after the change, and the teacher data60C) registered in the learning data14A.

The user who operates the input unit22may input a change instruction to change the output data82into a negative example. In this case, the reception unit12I of the learning unit12C receives the change instruction indicating negative example information from the input unit22via the acquisition unit12A.

The change unit12J of the learning unit12C then changes the output data82corresponding to the input data80including the teacher try-on person image40in the learning data14A into the output data82indicating the negative example (refer to the teacher data60B inFIG.4). In a case in which the output data82represents the negative example, an answer derived from the corresponding input data80is the negative example. In this way, the change unit12J may change the output data82into the negative example in accordance with the operation instruction on the input unit22made by the user.

In this case, the model learning unit12E can learn the learning model by using the teacher data60(including the teacher data60A, the teacher data60A after the change, the teacher data60B, and the teacher data60C) registered in the learning data14A.

As described above, the terminal device26generates and displays the virtual try-on image using the learning model generated by the information processing device10. As described above, the acquisition unit12A of the information processing device10may acquire, from the terminal device26, the display result information indicating the display result of the virtual try-on image displayed by the terminal device26.

In this case, the learning unit12C of the information processing device10may register a new piece of the teacher data60in the learning data14A using the display result information acquired from the terminal device26.

The display result information is information indicating the display result of the virtual try-on image by the terminal device26. In the present embodiment, the display result information represents first display result information or second display result information.

The first display result information includes positive example information indicating a positive example, the try-on person image51, and the output data82. The positive example information included in the first display result information is information indicating that the output data82included in the first display result information is the positive example. The try-on person image51included in the first display result information is the try-on person image50used for the virtual try-on image that is composited and displayed by using the output data82included in the first display result information by the terminal device26.

That is, the first display result information is information including the try-on person image51and the output data82used for displaying the virtual try-on image by the terminal device26, and the positive example information indicating that the output data82is the positive example.

The second display result information includes the negative example information indicating the negative example, the try-on person image51, and the output data82. The negative example information included in the second display result information is information indicating that the output data82included in the second display result information is the negative example. The try-on person image51included in the second display result information is the try-on person image50used for the virtual try-on image that is composited and displayed by using the output data82included in the second display result information by the terminal device26.

That is, the second display result information is information including the try-on person image51and the output data82used for displaying the virtual try-on image by the terminal device26, and the negative example information indicating that the output data82is the negative example.

In a case in which the acquisition unit12A acquires the first display result information, the model learning unit12E of the learning unit12C registers, in the learning data14A, a new piece of the teacher data60obtained by associating the input data80including the try-on person image51included in the first display result information with the output data82included in the first display result information.

Accordingly, as illustrated inFIG.4, a new piece of the teacher data60D obtained by associating the input data80including the try-on person image51included in the first display result information with the output data82included in the first display result information is registered in the learning data14A. The teacher data60D is an example of the teacher data60.

On the other hand, in a case in which the acquisition unit12A acquires the second display result information, the model learning unit12E of the learning unit12C registers, in the learning data14A, a new piece of the teacher data60obtained by associating the input data80including the try-on person image51included in the second display result information with the output data82including the negative example information indicating the negative example.

Accordingly, as illustrated inFIG.4, a new piece of the teacher data60E obtained by associating the input data80including the try-on person image51included in the second display result information with the output data82including the negative example information is registered in the learning data14A. The teacher data60E is an example of the teacher data60.

In this case, the model learning unit12E can learn the learning model by using the teacher data60(including the teacher data60A, the teacher data60A after the change, the teacher data60B, the teacher data60C, the teacher data60D, and the teacher data60E) registered in the learning data14A.

Returning toFIG.3, the description will be continued. Next, the following describes the deriving unit12D. When the acquisition unit12A acquires the try-on person image51from the terminal device26, the deriving unit12D derives the output data82of the try-on person represented by the try-on person image51using the try-on person image51and the learning model generated by the learning unit12C.

The deriving unit12D derives the output data82corresponding to the try-on person image51by inputting, as the input data80, the try-on person image51acquired from the terminal device26to the learning model generated by the learning unit12C.

The deriving unit12D then transmits the derived output data82to the terminal device26via the communication unit12B. Thus, the information processing device10can transmit, to the terminal device26, the output data82(the body shape parameter72, the compositing position information56, and the output data82) for the try-on person image51as a virtual try-on target received from the terminal device26.

Next, the following describes a function of the terminal device26.

The terminal device26includes a photographing unit26A, an input unit26B, a display unit26C, a storage unit26D, and a control unit26E. The photographing unit26A, the input unit26B, the display unit26C,26D, and the control unit26E are connected to each other to be able to exchange data and signals. The storage unit26D stores various kinds of data.

The control unit26E includes a communication unit26F, an acquisition unit26G, a generation unit26H, a display control unit26I, and a reception unit26J. Some of or all of the control unit26E, the communication unit26F, the acquisition unit26G, the generation unit26H, the display control unit26I, and the reception unit26J may be implemented by causing a processing device such as a CPU to execute a computer program, that is, by software, may be implemented by hardware such as an IC, or may be implemented by using both of software and hardware, for example.

The communication unit26F communicates with the information processing device10and the external server28via the communication line34.

The display control unit26I generates various images on the display unit26C. The reception unit26J receives, from the input unit26B, various kinds of information input by an operation instruction on the input unit26B made by the user. The acquisition unit26G acquires the output data82from the information processing device10via the communication unit26F and the communication line34.

The generation unit26H generates a virtual try-on image by compositing the try-on person image51and the clothing image42.

For example, it is assumed that the clothing ID of the clothing image42as the virtual try-on target is input by an operation instruction on the input unit26B made by the user. For example, the generation unit26H transmits a request for viewing the clothing DB14B to the information processing device10via the communication unit26F. The generation unit26H then receives the clothing DB14B from the information processing device10via the acquisition unit26G and the communication unit26F. The display control unit26I then displays a list of clothing images indicated by the received clothing DB14B on the display unit26C.

The user selects a clothing image as a try-on target by operating the input unit26B while referring to the list of clothing images displayed on the display unit26C. The reception unit26J then receives the clothing ID of the selected clothing image from the input unit26B. The generation unit26H then transmits the received clothing ID to the information processing device10via the communication unit26F.

The communication unit12B of the information processing device10reads, from the clothing DB14B, the clothing image42corresponding to the clothing ID received from the terminal device26to be transmitted to the terminal device26. Through this processing, the generation unit26H of the terminal device26acquires the clothing image42selected by the user from the information processing device10via the acquisition unit26G and the communication unit26F.

The generation unit26H also acquires the try-on person image51from the photographing unit26A via the acquisition unit26G. For example, the user inputs a photographing instruction for the photographing unit26A by operating the input unit26B. The photographing unit26A then acquires the try-on person image51by photographing to be output to the control unit26E.

The generation unit26H of the control unit26E acquires the try-on person image51as the virtual try-on target from the photographing unit26A via the acquisition unit26G. The generation unit26H may also acquire the try-on person image51from the storage unit26D.

There may be a case in which the try-on person image51acquired from the photographing unit26A or the storage unit26D is an image not including a clothing wearing region.

The clothing wearing region represents a target region of the try-on person at which the clothing image42as the virtual try-on target is worn. For example, in a case in which the clothing image42is a hat, the clothing wearing region is a head part of the try-on person. In a case in which the clothing image42is an upper garment, the clothing wearing region is an upper body of the try-on person.

In this case, the generation unit26H complements the try-on person image51so that the acquired try-on person image51becomes an image including the clothing wearing region. The generation unit26H may also complement the acquired try-on person image51to be an image including the clothing wearing region by changing a photographing angle of the try-on person represented by the try-on person image51. The generation unit26H may employ the complemented try-on person image51as the try-on person image51used for virtual try-on.

For the complement, a well-known method may be used. For example, the generation unit26H prepares a human body image having a standard human body shape in advance. The generation unit26H may use, as the complemented try-on person image51, a composite image obtained by superimposing the try-on person image51as an image not including the clothing wearing region on the human body image.

Next, the generation unit26H transmits the try-on person image51used for virtual try-on to the information processing device10via the communication unit26F. As described above, the deriving unit12D of the information processing device10then derives the output data82of the try-on person represented by the try-on person image51using the try-on person image51acquired from the terminal device26and the learning model generated by the learning unit12C.

The communication unit12B of the information processing device10transmits the output data82derived by the deriving unit12D to the terminal device26.

Accordingly, the generation unit26H of the terminal device26acquires the output data82from the information processing device10via the communication unit26F. In other words, the generation unit26H acquires the output data82of the try-on person represented by the try-on person image51that is derived by using the try-on person image51as the virtual try-on target and the learning model.

The generation unit26H then generates the virtual try-on image obtained by compositing the try-on person image51and the clothing image42using the try-on person image51as the virtual try-on target and the output data82of the try-on person represented by the try-on person image51that is acquired from the information processing device10.

Specifically, the generation unit26H generates the virtual try-on image obtained by compositing the clothing image42that is shape-modified in accordance with the body shape parameter72included in the output data82at the compositing position indicated by the compositing position information56included in the output data82that is derived by using the learning model in the try-on person image51.

The generation unit26H may generate the virtual try-on image similarly to generation of the superimposed image W performed by the learning unit12C of the information processing device10. That is, the generation unit26H enlarges, reduces, or shape-modifies the clothing image42as the virtual try-on target instructed by the user who operates the terminal device26to indicate the size of the body shape indicated by the body shape parameter72included in the derived output data82. For this processing, a well-known technique may be used.

The generation unit26H then generates the virtual try-on image by compositing the shape-modified clothing image42at the compositing position indicated by the compositing position information56included in the derived output data82. To perform compositing at the compositing position indicated by the compositing position information56, a well-known method may be used.

The display control unit26I displays the virtual try-on image generated by the generation unit26H on the display unit26C.

FIG.6is a schematic diagram illustrating an example of a virtual try-on image54displayed on the display unit26C of the terminal device26. As illustrated inFIG.6, the virtual try-on image54is displayed on the display unit26C.

As described above, the virtual try-on image54is the virtual try-on image54generated by the generation unit26H. Accordingly, on the display unit26C, displayed is the virtual try-on image54obtained by compositing the clothing image42that is shape-modified to have a body shape indicated by the body shape parameter72included in the output data82at the compositing position indicated by the compositing position information56included in the output data82that is derived by using the learning model. That is, on the display unit26C, the clothing image42that is enlarged, reduced, or shape-modified to match the body shape of the try-on person with high accuracy is superimposed on a try-on person region50A of the virtual try-on image54to be displayed.

Accordingly, the virtual try-on system1can easily provide, to the user, the virtual try-on image54that is composited with high accuracy. Specifically, the virtual try-on system1can easily provide the virtual try-on image54that is positioned with high accuracy and is corrected to match the body shape with high accuracy.

Next, the following describes a procedure of virtual try-on processing performed by the virtual try-on system1.

FIG.7is a flowchart illustrating an example of a procedure of information processing performed by the information processing device10.

First, the acquisition unit12A determines whether a photographing instruction is received from the input unit22(Step S100). For example, at the time of registering the teacher data60A in the information processing device10, the user facing the front of the display unit18operates the input unit22to input the photographing instruction. Accordingly, the acquisition unit12A receives the photographing instruction from the input unit22.

If positive determination is made at Step S100(Yes at Step S100), the process proceeds to Step S102. At Step S102, the photographing unit20and the three-dimensional data acquisition unit21photograph the try-on person, and obtains the teacher try-on person image40of the try-on person and the three-dimensional data of the try-on person. The acquisition unit12A then acquires the teacher try-on person image40from the photographing unit20(Step S104). The acquisition unit12A also acquires the three-dimensional data from the three-dimensional data acquisition unit21(Step S106).

Next, the acquisition unit12A acquires the characteristic information70from the input unit22(Step S108). For example, the try-on person facing the front of the display unit18operates the input unit22to input the characteristic information70. The input unit22then acquires the characteristic information70of the try-on person represented by the teacher try-on person image40acquired at Step S104.

Next, the model learning unit12E of the learning unit12C generates the output data82for the teacher data60A using the teacher try-on person image40, the three-dimensional data, and the characteristic information70that are acquired at Step S104to Step S108(Step S110).

Specifically, the model learning unit12E derives the body shape parameter72by introducing the three-dimensional data acquired at Step S106into a function or a model generated in advance. For example, the model learning unit12E derives, as the compositing position information56, a region having a rectangular shape surrounding the try-on person region40A representing the try-on person in the teacher try-on person image40acquired at Step S104(also refer toFIG.4).

The model learning unit12E then generates, as the output data82, the derived body shape parameter72and compositing position information56, and the characteristic information70acquired at Step S108.

Next, the model learning unit12E registers, in the learning data14A, the teacher data60A including the input data80that includes the teacher try-on person image40and the characteristic information70acquired at Step S104and Step S108, and the output data82(the body shape parameter72, the compositing position information56, and the characteristic information70) that is generated by using the input data80and the three-dimensional data (Step S112).

Next, the display control unit12H generates the superimposed image W using the teacher data60A registered at Step S112(Step S114). At Step S114, the display control unit12H generates the superimposed image W obtained by superimposing the clothing image42that is shape-modified in accordance with the body shape parameter72included in the output data82on the compositing position indicated by the compositing position information56included in the output data82included in the teacher data60A on the teacher try-on person image40included in the teacher data60A. The display control unit12H then displays the superimposed image W generated at Step S114on the display unit18(Step S116).

Through the processing at Step S114and Step S116, it is possible to provide the superimposed image W that is composited to match movement or a posture of the try-on person at the time of generating the teacher data60.

Next, the change unit12J determines whether a change instruction is received from the input unit22via the acquisition unit12A (Step S118). If negative determination is made at Step S118(No at Step S118), this routine is ended. If positive determination is made at Step S118(Yes at Step S118), the process proceeds to Step S120.

At Step S120, the change unit12J changes the output data82included in the teacher data60A registered at Step S112into information after the change indicated by the change instruction received at Step S118(Step S120). Through the processing at Step S120, at least part of the information of the output data82included in the teacher data60A is changed into change content instructed by the user. This routine is then ended.

On the other hand, if negative determination is made at Step S100described above (No at Step S100), the process proceeds to Step S122. At Step S122, the model learning unit12E determines whether to generate the learning model (Step S122). For example, the model learning unit12E determines to generate the learning model in a case in which the learning data14A is updated. “The learning data14A is updated” means a case in which at least part of the teacher data60registered in the learning data14A is changed, or a case in which a new piece of the teacher data60is registered in the learning data14A.

If positive determination is made at Step S122(Yes at Step S122), the process proceeds to Step S124. At Step S124, the model learning unit12E reads the learning data14A stored in the storage unit14(Step S124).

Next, the model learning unit12E learns the learning model by machine learning using the learning data14A read at Step S124(Step S126). The model learning unit12E then stores the learning model learned at Step S126in the storage unit14(Step S128). There may be a case in which a learning model has been already stored in the storage unit14. In this case, the model learning unit12E may delete the learning model that has been already stored from the storage unit14, and store the latest learning model that has been learned in the storage unit14. This routine is then ended.

On the other hand, if negative determination is made at Step S122described above (No at Step S122), the process proceeds to Step S130.

At Step S130, the communication unit12B determines whether the clothing ID is acquired from the terminal device (Step S130). If negative determination is made at Step S130(No at Step S130), the process proceeds to Step S136described later. If positive determination is made at Step S130(Yes at Step S130), the process proceeds to Step S132. At Step S132, the communication unit12B reads the clothing image42corresponding to the clothing ID acquired at Step S130from the clothing DB14B (Step S132). The communication unit12B then transmits the clothing image42read at Step S132to the terminal device26as an acquisition source of the clothing ID (Step S134).

Next, the deriving unit12D determines whether the try-on person image51is acquired from the terminal device26via the communication unit12B (Step S136). If positive determination is made at Step S136(Yes at Step S136), the process proceeds to Step S138.

At Step S138, the output data82is derived by introducing the try-on person image51acquired at Step S136into the learning model stored in the storage unit14(Step S138).

Next, the deriving unit12D transmits the output data82derived at Step S138to the terminal device26as a transmission source of the try-on person image51acquired at Step S136via the communication unit12B (Step S140). This routine is then ended.

If negative determination is made at Step S136described above (No at Step S136), the process proceeds to Step S142. At Step S142, the model learning unit12E determines whether the first display result information is acquired from the terminal device26via the acquisition unit12A (Step S142). If negative determination is made at Step S142(No at Step S142), the process proceeds to Step S146described later. If positive determination is made at Step S142(Yes at Step S142), the process proceeds to Step S144.

At Step S144, the model learning unit12E registers, in the learning data14A, the input data80including the try-on person image51included in the first display result information acquired at Step S142, the output data82included in the first display result information, and the teacher data60D (Step S144).

Next, the model learning unit12E determines whether the second display result information is acquired from the terminal device26via the acquisition unit12A (Step S146). If negative determination is made at Step S146(No at Step S146), this routine is ended. If positive determination is made at Step S146(Yes at Step S146), the process proceeds to Step S148. At Step S148, the model learning unit12E registers, in the learning data14A, a new piece of the teacher data60E obtained by associating the input data80including the try-on person image51included in the second display result information acquired at Step S146with the output data82including the negative example information indicating the negative example (Step S148). This routine is then ended.

Next, the following describes an example of a procedure of information processing performed by the terminal device26.FIG.8is a flowchart illustrating an example of the procedure of the information processing performed by the terminal device26.

First, the reception unit26J receives the clothing ID of the clothing image42as the virtual try-on target from the input unit26B (Step S200). The generation unit26H then transmits the clothing ID received at Step S200to the information processing device10via the communication unit26F (Step202). The generation unit26H acquires the clothing image42corresponding to the clothing ID transmitted at Step S200from the information processing device10(Step S204).

Next, the reception unit26J repeats negative determination (No at Step S206) until it is determined that the photographing instruction is received from the input unit26B (Yes at Step S206). If it is determined that the photographing instruction is received (Yes at Step S206), the process proceeds to Step S208.

At Step S208, the photographing unit26A photographs the try-on person, and obtains the try-on person image51(Step S208). The acquisition unit26G then acquires the try-on person image51from the photographing unit26A (Step S210).

Next, the generation unit26H determines whether the try-on person image51acquired at Step S210includes the clothing wearing region for the clothing image42acquired at Step S204(Step S212).

If it is determined that the clothing wearing region is not included (No at Step S212), the process proceeds to Step S214. At Step S214, the generation unit26H complements the try-on person image51acquired at Step S210to be an image including the clothing wearing region for the clothing image42acquired at Step S204(Step S214). In this case, the generation unit26H employs the complemented try-on person image51as the try-on person image51used for virtual try-on.

Next, the generation unit26H transmits the try-on person image51acquired at Step S210or the try-on person image51complemented at Step S214to the information processing device10via the communication unit26F (Step S216). As described above, the deriving unit12D of the information processing device10then derives the output data82of the try-on person represented by the try-on person image51using the try-on person image51acquired from the terminal device26and the learning model generated by the learning unit12C, and transmits the output data82to the terminal device26.

As such, the generation unit26H of the terminal device26acquires the output data82from the information processing device10via the communication unit26F (Step S218). In other words, the generation unit26H acquires the output data82of the try-on person represented by the try-on person image51that is derived by using the try-on person image51as the virtual try-on target and the learning model.

The generation unit26H then generates the virtual try-on image54by compositing the try-on person image51and the clothing image42by using the try-on person image51as the virtual try-on target transmitted to the information processing device10at Step S216and the output data82of the try-on person represented by the try-on person image51that is acquired from the information processing device10(Step S220).

Next, the display control unit26I displays the virtual try-on image54generated at Step S220on the display unit26C (Step S222). Accordingly, the virtual try-on image54with high accuracy is displayed on the display unit26C of the terminal device26.

Next, the reception unit26J determines whether error information is received from the input unit26B (Step S224).

There may be a case in which the user who operates the terminal device26feels a sense of incongruity for the virtual try-on image54displayed on the display unit26C. For example, there may be a case in which a positional relation between the try-on person image51and the clothing image42, the size of the displayed clothing image42, and the like in the virtual try-on image54are different from those imagined by the user. In such a case, the user operates the input unit26B to input error information for causing the output data82used for generating the virtual try-on image54to be the negative example. The reception unit26J then receives the error information from the input unit22(Yes at Step S224).

If positive determination is made at Step S224(Yes at Step S224), the process proceeds to Step S226. At Step S226, the reception unit26J includes the negative example information indicating the negative example, the try-on person image51acquired at Step S210, and the output data82acquired at Step S218. The second display result information is transmitted to the information processing device10via the communication unit26F (Step S226). The process then proceeds to Step S228. If negative determination is made at Step S224(No at Step S224), the process proceeds to Step S228.

At Step S228, the reception unit26J determines whether a correction instruction is received from the input unit22(Step S228).

For example, there may be a case in which the user who operates the terminal device26desires to correct the compositing position of the clothing image42, or correct the size of the clothing image42for the virtual try-on image54displayed on the display unit26C. In such a case, the user operates the input unit22to input the correction instruction for at least one of the body shape parameter72and the compositing position information56. There may also be a case in which the user who operates the terminal device26does not desire to make a change such as correction for the virtual try-on image54displayed on the display unit26C. In such a case, the user operates the input unit22to input an instruction indicating no correction.

At Step S228, the reception unit26J then determines whether the correction instruction is received from the input unit22. If the reception unit26J receives the correction instruction from the input unit22(Yes at Step S228), the process proceeds to Step S230.

At Step S230, the reception unit26J transmits, to the information processing device10via the communication unit26F, the first display result information including the positive example information indicating the positive example, the try-on person image51, and the output data82corrected to be correction content indicated by the correction instruction received at Step S228(Step S230). This try-on person image51is the try-on person image51that is used for the virtual try-on image54displayed on the display unit26C at Step S222. This routine is then ended.

On the other hand, if the reception unit26J receives the instruction indicating no correction from the input unit26B (No at Step S228), the process proceeds to Step S232. At Step S232, the reception unit26J transmits, to the information processing device10via the communication unit26F, the first display result information including the positive example information indicating the positive example, the try-on person image51, and the output data82acquired at Step S218(Step S232). This try-on person image51is the try-on person image51that is used for the virtual try-on image54displayed on the display unit26C at Step S222. This routine is then ended.

As described above, the virtual try-on system1according to the present embodiment includes the learning unit12C, the acquisition unit12A, the deriving unit12D, and the generation unit26H. The learning unit12C learns the learning model using the teacher try-on person image40as the input data80, and using the body shape parameter72indicating the body shape of the try-on person represented by the teacher try-on person image40and the compositing position information56of the clothing image42in the teacher try-on person image40as the output data82by machine learning using the three-dimensional data of the try-on person. The acquisition unit26G acquires the try-on person image51. The deriving unit12D derives the output data82of the try-on person represented by the try-on person image51using the try-on person image51and the learning model. The generation unit26H generates the virtual try-on image54by compositing the try-on person image51and the clothing image42using the derived output data82.

In this way, the virtual try-on system1according to the present embodiment derives the output data82(the body shape parameter72and the compositing position information56) of the try-on person represented by the try-on person image51by using the learning model learned by machine learning using the three-dimensional data of the try-on person. The derived output data82is then used to generate the virtual try-on image54by compositing the try-on person image51and the clothing image42.

Accordingly, the virtual try-on system1according to the present embodiment is not required to acquire information about the three-dimensional shape of the try-on person at the time of displaying the virtual try-on image54, and can generate the virtual try-on image54by compositing the try-on person image51and the clothing image42by using the body shape parameter72and the compositing position information56that are derived by using the learning model.

Thus, the virtual try-on system1according to the present embodiment can easily provide the virtual try-on image54that is composited with high accuracy.

Modification The embodiment described above describes a case in which the information processing device10includes the learning unit12C and the deriving unit12D, and the terminal device26includes the generation unit26H by way of example. However, the information processing device10may have a configuration further including at least one of the acquisition unit26G, the generation unit26H, the reception unit26J, and the display control unit26I. Alternatively, the terminal device26may have a configuration further including at least one of the acquisition unit12A, the learning unit12C, and the deriving unit12D.

In this case, with one information processing device10or one terminal device26, it is possible to generate the learning data14A, learn the learning model, derive the output data82of the try-on person represented by the try-on person image51, and generate and display the virtual try-on image54.

Next, the following describes a hardware configuration of the information processing device10, the terminal device26, and the external server28according to the present embodiment.FIG.9is a block diagram illustrating a hardware configuration example of the information processing device10, the terminal device26, and the external server28according to the present embodiment.

The information processing device10, the terminal device26, and the external server28according to the present embodiment have a hardware configuration utilizing a normal computer in which a display unit91, a communication I/F unit93, an input unit94, a CPU86, a read only memory (ROM)88, a random access memory (RAM)90, an HDD92, and the like are connected to each other via a bus96.

The CPU86is an arithmetic unit that controls processing performed by each of the information processing device10, the terminal device26, and the external server28according to the present embodiment. The RAM90stores data required for various kinds of processing performed by the CPU86. The ROM88stores a computer program and the like for implementing various kinds of processing performed by the CPU86. The HDD92stores data to be stored in the storage unit14and the storage unit26D described above. The communication I/F unit93is an interface connected to an external device or an external terminal via a communication line and the like for transmitting/receiving data to/from the connected external device or external terminal. The display unit91corresponds to each of the display unit18and the display unit26C described above. The input unit94receives an operation instruction from the user. The input unit94corresponds to the input unit22and the input unit26B described above.

A computer program for executing the various kinds of processing performed by the information processing device10, the terminal device26, and the external server28according to the present embodiment is embedded and provided in the ROM88and the like.

The computer program executed by the information processing device10, the terminal device26, and the external server28according to the present embodiment may be configured to be recorded and provided in a computer-readable recording medium such as a compact disc read only memory (CD-ROM), a flexible disk (FD), a compact disc recordable (CD-R), and a digital versatile disc (DVD), as an installable or executable file for these devices.

The computer program executed by the information processing device10, the terminal device26, and the external server28according to the present embodiment may be configured to be stored in a computer connected to a network such as the Internet and provided by being downloaded via the network. Furthermore, the computer program for executing pieces of the processing performed by the information processing device10, the terminal device26, and the external server28according to the present embodiment may be configured to be provided or distributed via a network such as the Internet.

In the computer program for executing pieces of the processing performed by the information processing device10, the terminal device26, and the external server28according to the present embodiment, the respective parts described above are generated on a main storage device.

The various kinds of information stored in the HDD92may also be stored in an external device. In this case, the external device and the CPU86may be configured to be connected via a network and the like.

While certain embodiments have been described, these embodiments have been presented by way of example only, and are not intended to limit the scope of the inventions. Indeed, the novel embodiments described herein may be embodied in a variety of other forms; furthermore, various omissions, substitutions and changes in the form of the embodiments described herein may be made without departing from the spirit of the inventions. The accompanying claims and their equivalents are intended to cover such forms or modifications as would fall within the scope and spirit of the inventions.