Patent Publication Number: US-2022215203-A1

Title: Storage medium, information processing apparatus, and determination model generation method

Description:
CROSS-REFERENCE TO RELATED APPLICATION 
     This application is based upon and claims the benefit of priority of the prior Japanese Patent Application No. 2021-937, filed on Jan. 6, 2021, the entire contents of which are incorporated herein by reference. 
     FIELD 
     The embodiment discussed herein is related to a storage medium, an information processing apparatus, and a determination model generation method. 
     BACKGROUND 
     In recent years, when an image that matches contents of a sentence (hereinafter, also referred to as a character string) or a sentence that matches contents of an image is searched for, a machine learning model (hereinafter, also referred to as a determination model) that determines a degree of matching between an image and a sentence may be used. 
     Such a determination model calculates a degree of matching between an image and a sentence by, for example, learning not only a pair of an image and sentence whose contents correspond to each other, but also a pair obtained by replacing one of the image and the sentence with another sample. 
     Jiasen Lu, Dhruv Batra, Devi Parikh, Stefan Lee, “ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks”, URL: https://arxiv.org/pdf/1908.02265.pdf is disclosed as related art. 
     SUMMARY 
     According to an aspect of the embodiments, a non-transitory computer-readable storage medium storing a determination model generation program that causes at least one computer to execute a process, the process includes generating, based on first training data in which image data and character string data that corresponds to the image data are associated with each other, second training data by replacing one of data included in the first training data selected from the image data and the character string data with another data; and generating, by using the first training data and the second training data as input data, a determination model that outputs information that indicates which training data selected from the first training data and the second training data is training data in which correspondence between the image data and the character string data is correct. 
     The object and advantages of the invention will be realized and attained by means of the elements and combinations particularly pointed out in the claims. 
     It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory and are not restrictive of the invention. 
    
    
     
       BRIEF DESCRIPTION OF DRAWINGS 
         FIG. 1  is a diagram illustrating a configuration of an information processing system  10 ; 
         FIG. 2  is a diagram illustrating a determination model MD11 in a first comparative example; 
         FIG. 3  is a diagram illustrating the determination model MD11 in the first comparative example; 
         FIG. 4  is a diagram illustrating a determination model MD12 in a second comparative example; 
         FIG. 5  is a diagram illustrating the determination model MD12 in the second comparative example; 
         FIG. 6  is a diagram illustrating a hardware configuration of an information processing apparatus  1 ; 
         FIG. 7  is a block diagram of functions of the information processing apparatus  1 ; 
         FIG. 8  is a flowchart illustrating an outline of determination model generation processing in a first embodiment; 
         FIG. 9  is a diagram illustrating processing of a determination model MD1 in the first embodiment; 
         FIG. 10  is a diagram illustrating processing of the determination model MD1 in the first embodiment; 
         FIG. 11  is a flowchart illustrating details of the determination model generation processing in the first embodiment; 
         FIG. 12  is a flowchart illustrating the details of the determination model generation processing in the first embodiment; 
         FIG. 13  is a flowchart illustrating the details of the determination model generation processing in the first embodiment; 
         FIG. 14  is a flowchart illustrating the details of the determination model generation processing in the first embodiment; 
         FIG. 15  is a flowchart illustrating the details of the determination model generation processing in the first embodiment; 
         FIG. 16  is a flowchart illustrating the details of the determination model generation processing in the first embodiment; 
         FIG. 17  is a flowchart illustrating the details of the determination model generation processing in the first embodiment; 
         FIG. 18  is a flowchart illustrating the details of the determination model generation processing in the first embodiment; 
         FIG. 19  is a diagram illustrating processing of a determination model MD2 in the first embodiment; and 
         FIG. 20  is a diagram illustrating the processing of the determination model MD2 in the first embodiment. 
     
    
    
     DESCRIPTION OF EMBODIMENTS 
     Here, the determination model as described above is trained by inputting a plurality of pairs of sentences and images at the same time, for example. With this configuration, the determination model may proceed with learning about a relationship between pieces of data, as compared with a case where, for example, the determination model is trained by inputting a pair of a sentence and an image one by one. 
     However, for example, in a case where learning is performed by using a plurality of pairs in which both images and sentences are randomly determined, the determination model may not be able to sufficiently learn about characteristics of each pair, and may not be able to calculate a degree of matching between an image and a sentence with sufficient accuracy. 
     Therefore, in one aspect, an embodiment aims to provide a determination model generation program, an information processing apparatus, and a determination model generation method that enable generation of a determination model that accurately determines correspondence between an image and a sentence. 
     According to one aspect, it is possible to generate a determination model that accurately determines correspondence between an image and a sentence. 
     [Configuration of Information Processing System] 
     First, a configuration of an information processing system  10  will be described.  FIG. 1  is a diagram illustrating the configuration of the information processing system  10 . 
     The information processing system  10  illustrated in  FIG. 1  includes, for example, an information processing apparatus  1  including one or more physical machines, and an operation terminal  2  in which an operator who generates a determination model (hereinafter, also simply referred to as an operator) performs input of necessary information, or the like. The operation terminal  2  may be, for example, a personal computer (PC). Furthermore, the information processing apparatus  1  and the operation terminal  2  are connected via a network NW such as the Internet, for example. 
     The information processing apparatus  1  generates a determination model by using, for example, a plurality of pieces of training data prepared in advance. Hereinafter, determination models in comparative examples will be described. 
     Determination Model in First Comparative Example 
       FIGS. 2 and 3  are diagrams illustrating a determination model MD11 in a first comparative example. For example,  FIG. 2  is a diagram illustrating processing of the determination model MD11 in a learning stage. Furthermore,  FIG. 3  is a diagram illustrating processing of the determination model MD11 in an inference stage. A first neural network NN 1  may be, for example, Transformer. Furthermore, a second neural network NN 2  may be, for example, a liner network for learning a score function. 
     Note that, hereinafter, description will be made assuming that each determination model includes the neural network NN 1  (hereinafter, also referred to as the first neural network NN 1 ) and the neural network NN 2  (hereinafter, also referred to as the second neural network NN 2 ). 
     First, the processing of the determination model MD11 in the learning stage will be described. 
     As illustrated in  FIG. 2 , the information processing apparatus  1  inputs, to the first neural network NN 1 , a pair of image data IM 1  and character string data ST 1  that are included in training data for learning. For example, the information processing apparatus  1  inputs, to the first neural network NN 1 , a pair of image data IM 1  and character string data ST 1  whose contents correspond to each other, or a pair of image data IM 1  and character string data ST 1  whose contents do not correspond. 
     Then, in response to reception of input of the pair of image data IM 1  and character string data ST 1 , the first neural network NN 1  calculates and outputs a vector indicating characteristics of the image data IM 1  (hereinafter, also referred to as an image vector) and a vector indicating characteristics of the character string data ST 1  (hereinafter, also referred to as a character string vector). 
     Subsequently, for example, the information processing apparatus  1  calculates an element product of the image vector and the character string vector that are output from the first neural network NN 1 , and inputs the element product to the second neural network NN 2 . 
     Then, for example, in response to reception of input of the element product, the second neural network NN 2  calculates and outputs a degree of matching between the contents indicated by the image data IM 1  and the contents indicated by the character string data ST 1 . 
     Thereafter, for example, the information processing apparatus  1  adjusts weights of the first neural network NN 1  and the second neural network NN 2  such that an error between the degree of matching output by the second neural network and a value (correct data) indicating whether or not the contents indicated by the image data IM 1  and the contents indicated by the character string data ST 1  correspond to each other becomes small. 
     Next, the processing of the determination model MD11 in the inference stage will be described. 
     As illustrated in  FIG. 3 , for example, in a case where training data to be inferred (hereinafter, also referred to as new training data) is input via the operation terminal  2 , the information processing apparatus  1  inputs a pair of image data IM 2  and character string data ST 2  that are included in the new training data to the first neural network NN 1 . 
     Then, in response to reception of input of the pair of image data IM 2  and character string data ST 2 , the first neural network NN 1  calculates and outputs an image vector indicating characteristics of the image data IM 2  and a character string vector indicating characteristics of the character string data ST 2 . 
     Subsequently, for example, the information processing apparatus  1  calculates an element product of the image vector and the character string vector that are output from the first neural network NN 1 , and inputs the element product to the second neural network NN 2 . 
     Then, for example, in response to reception of input of the element product, the second neural network NN 2  calculates and outputs a degree of matching between contents indicated by the image data IM 2  and contents indicated by the character string data ST 2 . 
     Thereafter, the information processing apparatus  1  outputs, for example, the degree of matching output by the second neural network to the operation terminal  2  as a degree of matching between the image data IM 2  and the character string data ST 2  that are included in the new training data. 
     Here, for example, in a case where training is performed by inputting a pair of image data IM 1  and character string data ST 1  one by one, it is not possible for the determination model MD11 to totally optimize learning, for example, it is not possible to learn about a relationship between pieces of data. Therefore, for example, in a case where image data IM 2  and character string data ST 2  related to the same object are input in the inference stage, the determination model MD11 may output a high degree of matching even in a case where contents of the image data IM 2  and contents of the character string data ST 2  indicate different situations. 
     Determination Model in Second Comparative Example 
     Next, a determination model MD12 in a second comparative example will be described.  FIGS. 4 and 5  are diagrams illustrating the determination model MD12 in the second comparative example. For example,  FIG. 4  is a diagram illustrating processing of the determination model MD12 in a learning stage. Furthermore,  FIG. 5  is a diagram illustrating processing of the determination model MD12 in an inference stage. 
     Note that, hereinafter, image data IM 1   a , image data IM 1   b , and image data IM 1   c  are also collectively referred to simply as image data IM 1 , and character string data ST 1   a , character string data ST 1   b , and character string data ST 1   c  are also collectively referred to simply as character string data ST 1 . 
     First, the processing of the determination model MD12 in the learning stage will be described. 
     As illustrated in  FIG. 4 , the information processing apparatus  1  inputs, to the first neural network NN 1 , a plurality of pairs of image data IM 1  and character string data ST 1  that are included in training data for learning. For example, the information processing apparatus  1  inputs, to the first neural network NN 1 , one pair of image data IM 1  and character string data ST 1  whose contents correspond to each other, and a plurality of pairs of image data IM 1  and character string data ST 1  whose contents do not correspond. 
     For example, as illustrated in  FIG. 4 , the information processing apparatus  1  inputs, to the first neural network NN 1 , a plurality of pairs including a pair of image data IM 1   a  and character string data ST 1   a  whose contents correspond to each other, a pair of image data IM 1   b  and character string data ST 1   b  whose contents do not correspond, and a pair of image data IM 1   c  and character string data ST 1   c  whose contents do not correspond. 
     Then, in response to reception of input of the pairs of image data IM 1  and character string data ST 1 , the first neural network NN 1  calculates and outputs image vectors indicating characteristics of the image data IM 1  and character string vectors indicating characteristics of the character string data ST 1 . 
     Subsequently, for example, the information processing apparatus  1  calculates element products of the image vectors and the character string vectors that are output from the first neural network NN 1 , and inputs the element products to the second neural network NN 2 . 
     Then, for example, in a case where input of the plurality of element products is received, the second neural network NN 2  calculates and outputs, for each pair of image data IM 1  and character string data ST 1 , a degree of matching between the image data IM 1  and the character string data ST 1  that are included in each pair. 
     Thereafter, for example, the information processing apparatus  1  adjusts the weights of the first neural network NN 1  and the second neural network NN 2  such that a classification error (cross entropy) calculated from the plurality of degrees of matching output by the second neural network and information indicating the pair of image data IM 1  and character string data ST 1  whose contents correspond to each other (hereinafter, also referred to as a correct pair) becomes small. 
     Next, the processing of the determination model MD12 in the inference stage will be described. 
     As illustrated in  FIG. 5 , for example, in a case where new training data is input via the operation terminal  2 , the information processing apparatus  1  inputs, to the first neural network NN 1 , a pair of image data IM 2  and character string data ST 2  that are included in the new training data. 
     Then, in response to reception of input of the pair of image data IM 2  and character string data ST 2 , the first neural network NN 1  calculates and outputs an image vector indicating characteristics of the image data IM 2  and a character string vector indicating characteristics of the character string data ST 2 . 
     Subsequently, for example, the information processing apparatus  1  calculates an element product of the image vector and the character string vector that are output from the first neural network NN 1 , and inputs the element product to the second neural network NN 2 . 
     Then, for example, in response to reception of input of the element product, the second neural network NN 2  calculates and outputs a degree of matching between contents indicated by the image data IM 2  and contents indicated by the character string data ST 2 . 
     Thereafter, the information processing apparatus  1  outputs, for example, the degree of matching output by the second neural network to the operation terminal  2  as a degree of matching between the image data IM 2  and the character string data ST 2  that are included in the new training data. 
     For example, the determination model MD12 is trained by using the plurality of pairs of image data IM 1  and character string data ST 1  at the same time. With this configuration, the determination model MD12 may proceed with learning of a relationship between pieces of training data as compared with the determination model MD11 described with reference to  FIGS. 2 and 3 , and may perform learning that is totally optimized. 
     However, for example, in a case where training is performed by using a plurality of pairs in which both the image data IM 1  and the character string data ST 1  are randomly determined, it is not possible for the determination model MD12 to sufficiently learn about characteristics of each pair. Thus, it may not possible for the determination model MD12 to calculate the degree of matching between the image data IM 2  and the character string data ST 2  with sufficient accuracy in the inference stage. 
     Therefore, the information processing apparatus  1  in the present embodiment generates, on the basis of training data (hereinafter, also referred to as first training data) in which image data IM 1  and character string data ST 1  corresponding to the image data IM 1  are associated with each other, training data (hereinafter, also referred to as second training data) by replacing one of the image data IM 1  and the character string data ST 1  that are included in the first training data with another data. 
     Then, the information processing apparatus  1  generates, by using the first training data and the second training data as input data, a determination model that outputs information indicating which piece of training data in the first training data and the second training data is training data in which correspondence between the image data IM 1  and the character string data ST 1  is correct. 
     For example, the information processing apparatus  1  in the present embodiment generates, in a case where a plurality of pairs of image data IM 1  and character string data ST 1  to be input to the determination model is generated, the plurality of pairs by combining different character string data ST 1  with the same image data IM 1 . Furthermore, in this case, the information processing apparatus  1  generates the plurality of pairs by combining different image data IM 1  with the same character string data ST 1 . Then, the information processing apparatus  1  generates the determination model by causing the determination model to learn a degree of similarity calculated for each of the generated plurality of pairs. 
     With this configuration, the information processing apparatus  1  may cause a determination model to learn a more detailed relationship between pieces of training data, and may generate a determination model that accurately determines correspondence between image data IM 1  and character string data ST 1   
     [Hardware Configuration of Information Processing Apparatus] 
     Next, a hardware configuration of the information processing apparatus  1  will be described.  FIG. 6  is a diagram illustrating the hardware configuration of the information processing apparatus  1 . 
     As illustrated in  FIG. 6 , the information processing apparatus  1  includes a central processing unit (CPU)  101  as a processor, a memory  102 , a communication device  103 , and a storage medium  104 . Each of the units is connected to each other via a bus  105 . 
     The storage medium  104  includes, for example, a program storage area (not illustrated) that stores a program  110  for performing determination model generation processing. Furthermore, the storage medium  104  includes, for example, an information storage area  130  that stores information used when the determination model generation processing is performed. Note that the storage medium  104  may be, for example, a hard disk drive (HDD) or a solid state drive (SSD). 
     The CPU  101  executes the program  110  loaded from the storage medium  104  into the memory  102  to perform the determination model generation processing. 
     Furthermore, the communication device  103  communicates with the operation terminal  2  via the network NW, for example. 
     [Functions of Information Processing Apparatus] 
     Next, functions of the information processing apparatus  1  will be described.  FIG. 7  is a block diagram of the functions of the information processing apparatus  1 . 
     As illustrated in  FIG. 7 , the information processing apparatus  1  implements various functions including a data reception unit  111 , a data management unit  112 , a data generation unit  113 , a vector generation unit  114 , a similarity calculation unit  115 , a model training unit  116 , a matching degree calculation unit  117 , and a result output unit  118 , for example, by hardware such as the CPU  101  and the memory  102  organically cooperating with the program  110 . 
     Furthermore, as illustrated in  FIG. 7 , the information processing apparatus  1  stores, for example, first training data DT 1 , second training data DT 2 , and new training data DT 3  in the information storage area  130 . 
     First, the functions in a learning stage will be described. 
     The data reception unit  111  receives, for example, the first training data DT 1  transmitted from the operation terminal  2 . The first training data DT 1  is training data including image data IM 1  and character string data ST 1  whose contents correspond to each other. Then, the data management unit  112  stores, for example, the first training data DT 1  received by the data reception unit  111  in the information storage area  130 . 
     The data generation unit  113  generates the second training data DT 2  on the basis of the first training data DT 1  stored in the information storage area  130 . The second training data DT 2  is training data obtained by replacing one of the image data IM 1  and the character string data ST 1  that are included in the first training data DT 1  with another data. For example, the second training data DT 2  is training data including image data IM 1  and character string data ST 1  whose contents do not correspond. Then, the data management unit  112  stores, for example, the second training data DT 2  generated by the data generation unit  113  in the information storage area  130 . 
     The vector generation unit  114  generates an image vector indicating characteristics of each of the image data IM 1  included in the first training data DT 1  and the second training data DT 2  by inputting the first training data DT 1  and the second training data DT 2  that are stored in the information storage area  130  to the first neural network NN 1 . Furthermore, the vector generation unit  114  generates character string vectors indicating characteristics of the character string data ST 1  included in the first training data DT 1  and the second training data DT 2  by inputting the first training data DT 1  and the second training data DT 2  that are stored in the information storage area  130  to the first neural network NN 1 . 
     The similarity calculation unit  115  calculates, for example, an inner product of an image vector and a character string vector that are generated by the vector generation unit  114  from the first training data DT 1  as a degree of similarity between the image data IM 1  and the character string data ST 1  that are included in the first training data DT 1 . Furthermore, the similarity calculation unit  115  calculates, for example, an inner product of an image vector and a character string vector that are generated by the vector generation unit  114  from the second training data DT 2  as a degree of similarity between the image data IM 1  and the character string data ST 1  that are included in the second training data DT 2 . 
     The model training unit  116  calculates, for example, a classification error from each degree of similarity calculated by the similarity calculation unit  115  and information indicating the first training data DT 1  (hereinafter, also referred to as first information). Then, the model training unit  116  adjusts (learns) the weight of the first neural network NN 1  such that the calculated classification error becomes small. 
     The matching degree calculation unit  117  calculates, for example, an element product of an image vector and a character string vector that are generated by the vector generation unit  114  from the first training data DT 1 . Furthermore, the matching degree calculation unit  117  calculates, for example, an element product of an image vector and a character string vector that are generated by the vector generation unit  114  from the second training data DT 2 . For example, the matching degree calculation unit  117  calculates an element product for one piece of the first training data DT 1  and element products for a plurality of pieces of second training data DT 2 . 
     Then, the matching degree calculation unit  117  calculates, for example, a degree of matching between the image data IM 1  and the character string data ST 1  that are included in the first training data DT 1  by inputting the element product for the first training data DT 1  to the second neural network NN 2 . Furthermore, the matching degree calculation unit  117  calculates, for example, degrees of matching between the image data IM 1  and the character string data ST 1  that are included in the second training data DT 2  by inputting the element products for the second training data DT 2  to the second neural network NN 2 . 
     Moreover, for example, the model training unit  116  adjusts the weights of the first neural network NN 1  and the second neural network NN 2  such that an error between the degree of matching for the first training data DT 1  and a value corresponding to the first training data DT 1  (hereinafter, also referred to as second information) becomes small. Furthermore, for example, the model training unit  116  adjusts the weights of the first neural network NN 1  and the second neural network NN 2  such that an error between the degrees of matching for the second training data DT 2  and a value corresponding to the second training data DT 2  becomes small. 
     Next, the functions in an inference stage will be described. 
     The data reception unit  111  receives, for example, the new training data DT 3  transmitted from the operation terminal  2 . Then, the data management unit  112  stores, for example, the new training data DT 3  received by the data reception unit  111  in the information storage area  130 . 
     The vector generation unit  114  generates an image vector indicating characteristics of image data IM 2  included in the new training data DT 3  by inputting the new training data DT 3  received by the data reception unit  111  to the first neural network NN 1 . Furthermore, the vector generation unit  114  generates a character string vector indicating characteristics of character string data ST 2  included in the new training data DT 3  by inputting the new training data DT 3  to the first neural network NN 1 . 
     The similarity calculation unit  115  calculates, for example, an inner product of an image vector and a character string vector that are generated by the vector generation unit  114  from the new training data DT 3  as a degree of similarity between the image data IM 2  and the character string data ST 2  that are included in the new training data DT 3 . 
     The matching degree calculation unit  117  calculates, for example, an element product of an image vector and a character string vector that are generated by the vector generation unit  114  from the new training data DT 3 . 
     Then, the matching degree calculation unit  117  calculates, for example, a degree of matching between the image data IM 2  and the character string data ST 2  that are included in the new training data DT 3  by inputting the element product for the new training data DT 3  to the second neural network NN 2 . 
     The result output unit  118  outputs, for example, a degree of similarity calculated by the similarity calculation unit  115  to the operation terminal  2  as a degree of similarity between the image data IM 2  and the character string data ST 2  that are included in the new training data DT 3 . Furthermore, the result output unit  118  outputs, for example, a degree of matching calculated by the matching degree calculation unit  117  to the operation terminal  2  as a degree of matching between the image data IM 2  and the character string data ST 2  that are included in the new training data DT 3 . 
     Note that, hereinafter, a case where the matching degree calculation unit  117  calculates an element product of an image vector and a character string vector will be described. However, the matching degree calculation unit  117  may calculate, for example, the vector sum of an image vector and a character string vector. Then, in this case, the matching degree calculation unit  117  may input the calculated vector sum to the second neural network NN 2 . 
     Outline of First Embodiment 
     Next, an outline of a first embodiment will be described.  FIG. 8  is a flowchart illustrating an outline of determination model generation processing in the first embodiment. 
     As illustrated in  FIG. 8 , the information processing apparatus  1  stands by until, for example, a model generation timing comes (NO in S 11 ). The model generation timing may be, for example, a timing at which an operator inputs that a determination model is to be generated. 
     Then, in a case where the model generation timing comes (YES in S 11 ), the information processing apparatus  1  generates, on the basis of the first training data DT 1  in which the image data IM 1  and the character string data ST 1  corresponding to the image data IM 1  are associated with each other, the second training data DT 2  by replacing one of the image data IM 1  and the character string data ST 1  with another data (S 12 ). 
     Moreover, the information processing apparatus  1  generates, by using the first training data DT 1  and the second training data DT 2  as input data, a determination model that outputs information indicating which piece of training data in the first training data DT 1  and the second training data DT 2  is training data in which correspondence between the image data IM 1  and the character string data ST 1  is correct (S 13 ). 
     With this configuration, the information processing apparatus  1  may cause the determination model to learn a detailed relationship between pieces of the training data. Therefore, the information processing apparatus  1  may generate the determination model that accurately determines correspondence between the image data IM 1  and the character string data ST 1 . 
     Furthermore, by replacing one of the image data IM 1  and the character string data ST 1  that are included in the first training data DT 1  with another data to generate the second training data DT 2 , the information processing apparatus  1  may suppress increase in a work load and work time needed to generate the second training data DT 2 . 
     [Specific Example (1) of Determination Model] 
     Next, a determination model MD1 in the first embodiment will be described.  FIGS. 9 and 10  are diagrams illustrating processing of the determination model MD1 in the first embodiment. 
     First, processing of the determination model MD1 in a learning stage will be described. 
     As illustrated in  FIG. 9 , the information processing apparatus  1  inputs, to the first neural network NN 1 , a pair of image data IM 1  and character string data ST 1  that are included in the first training data DT 1  and pairs of image data IM 1  and character string data ST 1  that are included in the second training data DT 2 . 
     For example, as illustrated in  FIG. 9 , the information processing apparatus  1  inputs, to the first neural network NN 1 , a plurality of pairs including a pair of image data IM 1   a  and character string data ST 1   a  that are included in the first training data DT 1 , a pair of image data IM 1   b  and character string data ST 1   b  that are included in the second training data DT 2 , and a pair of image data IM 1   c  and character string data ST 1   c  that are included in the second training data DT 2 . 
     Then, for example, in response to reception of input of the pair of image data IM 1   a  and character string data ST 1   a , the first neural network NN 1  calculates and outputs an image vector indicating characteristics of the image data IM 1   a  and a character string vector indicating characteristics of the character string data ST 1   a . Similarly, for example, in response to reception of input of the pair of image data IM 1   b  and character string data ST 1   b , the first neural network NN 1  calculates and outputs an image vector indicating characteristics of the image data IM 1   b  and a character string vector indicating characteristics of the character string data ST 1   b . Moreover, for example, in response to reception of input of the pair of image data IM 1   c  and character string data ST 1   c , the first neural network NN 1  calculates and outputs an image vector indicating characteristics of the image data IM 1   c  and a character string vector indicating characteristics of the character string data ST 1   c.    
     Next, the information processing apparatus  1  calculates, for example, an inner product of the image vector corresponding to the image data IM 1   a  and the character string vector corresponding to the character string data ST 1   a  as a degree of similarity between the image data IM 1   a  and the character string data ST 1   a . Similarly, the information processing apparatus  1  calculates, for example, an inner product of the image vector corresponding to the image data IM 1   b  and the character string vector corresponding to the character string data ST 1   b  as a degree of similarity between the image data IM 1   b  and the character string data ST 1   b . Moreover, the information processing apparatus  1  calculates, for example, an inner product of the image vector corresponding to the image data IM 1   c  and the character string vector corresponding to the character string data ST 1   c  as a degree of similarity between the image data IM 1   c  and the character string data ST 1   c.    
     Thereafter, for example, the information processing apparatus  1  adjusts the weight of the first neural network NN 1  such that a classification error calculated from the degree of similarity for each of the first training data DT 1  and the second training data DT 2  and information specifying the pair (correct pair) of image data IM 1  and character string data ST 1  that are included in the first training data DT 1  becomes small. 
     Next, processing of the determination model MD1 in an inference stage will be described. 
     As illustrated in  FIG. 10 , the information processing apparatus  1  inputs, to the first neural network NN 1 , a pair of image data IM 2  and character string data ST 2  that are included in the new training data DT 3 . 
     Then, for example, in response to reception of input of the pair of image data IM 2  and character string data ST 2 , the first neural network NN 1  calculates and outputs an image vector indicating characteristics of the image data IM 2  and a character string vector indicating characteristics of the character string data ST 2 . 
     Thereafter, the information processing apparatus  1  calculates, for example, an inner product of the image vector corresponding to the image data IM 2  and the character string vector corresponding to the character string data ST 2  as a degree of similarity between the image data IM 2  and the character string data ST 2 . 
     Then, the information processing apparatus  1  outputs, for example, the calculated degree of similarity to the operation terminal  2  as a degree of similarity between the image data IM 2  and the character string data ST 2  that are included in the new training data DT 3 . 
     For example, unlike the case described with reference to  FIGS. 2 to 5 , the information processing apparatus  1  generates the determination model MD1 using only the first neural network NN 1  (determination model MD1 having no second neural network NN 2 ). 
     With this configuration, the information processing apparatus  1  may generate the first neural network NN 1  capable of outputting an image vector and character string vector different for each piece of training data. 
     Details of First Embodiment 
     Next, details of the first embodiment will be described.  FIGS. 11 to 18  are flowcharts illustrating details of the determination model generation processing in the first embodiment. Furthermore,  FIGS. 19 and 20  are diagrams illustrating the details of the determination model generation processing in the first embodiment. 
     [Data Management Processing] 
     First, in the determination model generation processing, processing of storing the first training data DT 1  in the information storage area  130  (hereinafter, also referred to as data management processing) will be described.  FIG. 11  is a flowchart illustrating the data management processing. 
     As illustrated in  FIG. 11 , for example, the data reception unit  111  stands by until receiving a plurality of pieces of training data DT 1  transmitted from the operation terminal  2  (NO in S 21 ). 
     Then, in the case of receiving the plurality of pieces of training data DT 1  (YES in S 21 ), the data management unit  112  stores the plurality of pieces of training data DT 1  received in the processing in S 21  in the information storage area  130  (S 22 ). 
     [Main Processing of Determination Model Generation Processing] 
     Next, main processing of the determination model generation processing will be described.  FIGS. 12 to 16  are flowcharts illustrating the main processing of the determination model generation processing. 
     As illustrated in  FIG. 12 , the data generation unit  113  stands by until a model generation timing comes (NO in S 31 ). 
     Then, in a case where the model generation timing comes (YES in S 31 ), the data generation unit  113  specifies any one piece of first training data DT 1  among the plurality of pieces of first training data DT 1  stored in the information storage area  130  (S 32 ). 
     Subsequently, the data generation unit  113  generates R pieces of second training data DT 2  by replacing one of image data IM 1  and character string data ST 1  that are included in the first training data DT 1  specified in the processing in S 32  with another data (S 33 ). Hereinafter, details of the processing in S 33  will be described. 
     [Details (1) of Processing in S 33 ] 
     First, a first example of the processing in S 33  will be described.  FIG. 15  is a flowchart illustrating the details of the processing in S 33 . 
     As illustrated in  FIG. 15 , for example, the data generation unit  113  duplicates R pieces of image data IM 1  included in the first training data DT 1  specified in the processing in S 32  (S 61 ). 
     Then, for example, from a plurality of pieces of character string data ST 1  included in the plurality of pieces of first training data DT 1  stored in the information storage area  130 , the data generation unit  113  specifies R pieces of character string data ST 1  other than character string data ST 1  included in the first training data DT 1  specified in the processing in S 32  (S 62 ). 
     Thereafter, for example, the data generation unit  113  generates R pieces of second training data DT 2  by associating the R pieces of image data IM 1  duplicated in the processing in S 61  with the R pieces of character string data ST 1  specified in the processing in S 62  (S 63 ). 
     [Details (2) of Processing in S 33 ] 
     Next, a second example of the processing in S 33  will be described.  FIG. 16  is another flowchart illustrating the details of the processing in S 33 . 
     For example, the data generation unit  113  duplicates R pieces of character string data ST 1  included in the first training data DT 1  specified in the processing in S 32  (S 71 ). 
     Then, for example, from a plurality of pieces of image data IM 1  included in the plurality of pieces of first training data DT 1  stored in the information storage area  130 , the data generation unit  113  specifies R pieces of image data IM 1  other than image data IM 1  included in the first training data DT 1  specified in the processing in S 32  (S 72 ). 
     Thereafter, for example, the data generation unit  113  generates R pieces of second training data DT 2  by associating the R pieces of character string data ST 1  duplicated in the processing in S 71  with the R pieces of image data IM 1  specified in the processing in S 72  (S 73 ). 
     For example, the information processing apparatus  1  may facilitate learning about a relationship between pieces of training data by generating a determination model by using a plurality of pieces of second training data DT 2  in which either image data IM 1  or character string data ST 1  is matched. 
     Therefore, the information processing apparatus  1  may generate the determination model that accurately determines correspondence between the image data IM 1  and the character string data ST 1 . 
     Note that, for example, the data generation unit  113  may generate a part of the R pieces of second training data DT 2  by performing the processing in S 61  to S 63 , and may generate another part of the R pieces of second training data DT 2  by performing the processing in S 71  to S 73 . 
     Furthermore, for example, the data generation unit  113  may generate the R pieces of second training data DT 2  by performing the processing in S 61  to S 63  and the processing in S 71  to S 73  in parallel. 
     Returning to  FIG. 12 , the vector generation unit  114  calculates, by using the first neural network NN 1 , an image vector for the image data IM 1  and a character string vector for the character string data ST 1  for each of the first training data DT 1  specified in the processing in S 32  and the R pieces of second training data DT 2  generated in the processing in S 33  (S 34 ). 
     For example, the vector generation unit  114  inputs, to the first neural network NN 1 , each of the first training data DT 1  specified in the processing in S 32  and the R pieces of second training data DT 2  generated in the processing in S 33 . Then, the vector generation unit  114  acquires each of the image vectors and the character string vectors that are output from the first neural network NN 1  as an image vector and a character string vector for each of the first training data DT 1  specified in the processing in S 32  and the R pieces of second training data DT 2  generated in the processing in S 33 . 
     Then, for each of the first training data DT 1  specified in the processing in S 32  and the R pieces of second training data DT 2  generated in the processing in S 33 , the similarity calculation unit  115  calculates a degree of similarity between the image data IM 1  and the character string data ST 1  by calculating an inner product of the vectors calculated in the processing in S 34  (S 35 ). 
     Subsequently, as illustrated in  FIG. 13 , the model training unit  116  adjusts the weight of the first neural network NN 1  such that a classification error calculated from each degree of similarity calculated in the processing in S 35  becomes small (S 41 ). 
     For example, the model training unit  116  adjusts the weight of the first neural network NN 1  such that a classification error L indicated in the following Equation (1) becomes small. 
     
       
         
           
             
               
                 
                   
                     [ 
                     
                       Expression 
                       ⁢ 
                       
                           
                       
                       ⁢ 
                       1 
                     
                     ] 
                   
                   ⁢ 
                   
                       
                   
                 
               
               
                 
                     
                 
               
             
             
               
                 
                   L 
                   = 
                   
                     
                       - 
                       log 
                     
                     ⁢ 
                     
                       
                         e 
                         
                           E 
                           p 
                         
                       
                       
                         
                           ∑ 
                           
                             i 
                             = 
                             0 
                           
                           R 
                         
                         ⁢ 
                         
                           e 
                           
                             E 
                             i 
                           
                         
                       
                     
                   
                 
               
               
                 
                   ( 
                   1 
                   ) 
                 
               
             
           
         
       
     
     In the above Equation (1), E i  indicates a degree of similarity corresponding to an i-th piece of training data among the first training data DT 1  specified in the processing in S 32  and the R pieces of second training data DT 2  generated in the processing in S 33 . Furthermore, E p  indicates a degree of similarity corresponding to the first training data DT 1  specified in the processing in S 32 . 
     Next, the matching degree calculation unit  117  specifies any one piece of second training data DT 2  included in the R pieces of second training data DT 2  generated in the processing in S 33  (S 42 ). 
     Then, for each of the first training data DT 1  specified in the processing in S 32  and the second training data DT 2  specified in the processing in S 42 , the matching degree calculation unit  117  calculates an element product of the vectors calculated in the processing in S 34  (S 43 ). 
     Moreover, the matching degree calculation unit  117  calculates, by using the second neural network NN 2 , a degree of matching between the image data IM 1  and the character string data ST 1  for each of an element product corresponding to the first training data DT 1  specified in the processing in S 32  and an element product corresponding to the second training data DT 2  specified in the processing in S 42  (S 44 ). 
     For example, the matching degree calculation unit  117  inputs each of the element products calculated in the processing in S 44  to the second neural network NN 2 . Then, the matching degree calculation unit  117  acquires the degrees of matching output from the second neural network NN 2  as a degree of matching between the image data IM 1  and the character string data ST 1  that are included in the first training data DT 1  specified in the processing in S 32  and a degree of matching between the image data IM 1  and the character string data ST 1  that are included in the second training data DT 2  specified in the processing in S 42 . 
     Subsequently, as illustrated in  FIG. 14 , the model training unit  116  calculates an error between the degree of matching for the first training data DT 1  specified in the processing in S 32  and a value corresponding to the first training data DT 1  (S 51 ). 
     Furthermore, the model training unit  116  calculates an error between the degree of matching for the second training data DT 2  specified in the processing in S 42  and a value corresponding to the second training data DT 2  (S 52 ). 
     Thereafter, the model training unit  116  adjusts the weights of the first neural network NN 1  and the second neural network NN 2  such that the errors calculated in the processing in S 51  and S 52  become small (S 53 ). 
     For example, in a case where the degree of matching for the first training data DT 1  specified in the processing in S 32  is a value between 0 and 1, and the value corresponding to the first training data DT 1  is 1, the model training unit  116  adjusts the weight of the second neural network NN 2  such that the degree of matching for the first training data DT 1  specified in the processing in S 32  approaches 1. Furthermore, the model training unit  116  adjusts the weights of the first neural network NN 1  and the second neural network NN 2  such that the error between the degree of matching for the second training data DT 2  specified in the processing in S 42  and the value corresponding to the second training data DT 2  becomes small. 
     Then, the model training unit  116  determines whether or not the first neural network NN 1  and the second neural network NN 2  satisfy a predetermined condition (S 54 ). 
     For example, the model training unit  116  calculates the sum of the error calculated in the processing in S 51  (error for the first training data DT 1 ) and the error calculated in the processing in S 52  (error for the second training data DT 2 ). Subsequently, the model training unit  116  calculates, for example, an average of the calculated sum and the classification error calculated from each degree of similarity calculated in the processing in S 35 . Then, for example, in a case where the calculated average is below a predetermined threshold, the model training unit  116  determines that the first neural network NN 1  and the second neural network NN 2  satisfy the predetermined condition. 
     As a result, in a case where it is determined that the first neural network NN 1  and the second neural network NN 2  do not satisfy the predetermined condition (NO in S 55 ), the information processing apparatus  1  performs the processing after S 32  again. 
     On the other hand, in a case where it is determined that the first neural network NN 1  and the second neural network NN 2  satisfy the predetermined condition (YES in S 55 ), the information processing apparatus  1  ends the main processing of the determination model generation processing. 
     [Data Estimation Processing (1)] 
     Next, in the determination model generation processing, a first example of processing of determining whether or not contents of image data IM 2  and contents of character string data ST 2  that are included in the new training data DT 3  match (hereinafter, also referred to as data estimation processing) will be described.  FIG. 17  is a flowchart illustrating the first example of the data estimation processing. 
     As illustrated in  FIG. 17 , for example, the data reception unit  111  stands by until receiving the new training data DT 3  transmitted from the operation terminal  2  (NO in S 81 ). 
     Then, in the case of receiving the new training data DT 3  (YES in S 81 ), the vector generation unit  114  calculates, by using the first neural network NN 1 , an image vector for the image data IM 2  and a character string vector for the character string data ST 2  that are included in the new training data DT 3  received in the processing in S 81  (S 82 ). 
     For example, the vector generation unit  114  inputs the new training data DT 3  to the first neural network NN 1 . Then, the vector generation unit  114  acquires the image vector and the character string vector that are output from the first neural network NN 1  as an image vector and a character string vector for the new training data DT 3 . 
     Then, for example, by calculating an inner product of the vectors calculated in the processing in S 82 , the similarity calculation unit  115  calculates a degree of similarity between the image data IM 2  and the character string data ST 2  that are included in the new training data DT 3  received in the processing in S 81  (S 83 ). 
     Thereafter, the result output unit  118  outputs, for example, the degree of similarity calculated in the processing in S 83  (S 84 ). 
     For example, the result output unit  118  outputs the degree of similarity calculated in the processing in S 83  to the operation terminal  2 . 
     Note that, for example, in a case where a degree of similarity for each of a plurality of pieces of new training data DT 3  is calculated in the processing in S 83 , the result output unit  118  may output information indicating each of the plurality of pieces of new training data DT 3  in descending order of the degree of similarity calculated in the processing in S 83 . 
     [Data Estimation Processing (2)] 
     Next, a second example of the data estimation processing will be described.  FIG. 18  is a flowchart illustrating the second example of the data estimation processing. 
     As illustrated in  FIG. 18 , for example, the data reception unit  111  stands by until receiving the new training data DT 3  transmitted from the operation terminal  2  (NO in S 91 ). 
     Then, in the case of receiving the new training data DT 3  (YES in S 91 ), the vector generation unit  114  calculates, by using the first neural network NN 1 , an image vector for the image data IM 2  and a character string vector for the character string data ST 2  that are included in the new training data DT 3  received in the processing in S 91  (S 92 ). 
     Then, the matching degree calculation unit  117  calculates an element product of the vectors calculated in the processing in S 92  (S 93 ). 
     Moreover, the matching degree calculation unit  117  calculates, by using the second neural network NN 2 , a degree of matching between the image data IM 2  and the character string data ST 2  that are included in the new training data DT 3  received in the processing in S 91  (S 94 ). 
     For example, the matching degree calculation unit  117  inputs the element product calculated in the processing in S 94  to the second neural network NN 2 . Then, the matching degree calculation unit  117  acquires the degree of matching output from the second neural network NN 2  as a degree of matching between the image data IM 2  and the character string data ST 2  that are included in the new training data DT 3  received in the processing in S 91 . 
     Thereafter, the result output unit  118  outputs, for example, the degree of matching calculated in the processing in S 94  (S 95 ). 
     Note that, for example, in a case where a degree of matching for each of a plurality of pieces of new training data DT 3  is calculated in the processing in S 94 , the result output unit  118  may output information indicating each of the plurality of pieces of new training data DT 3  in descending order of the degree of matching calculated in the processing in S 94 . 
     [Specific Example (2) of Determination Model] 
     Next, a determination model MD2 in the first embodiment will be described.  FIGS. 19 and 20  are diagrams illustrating processing of the determination model MD2 in the first embodiment. For example,  FIGS. 19 and 20  are diagrams illustrating the processing of the determination model corresponding to the contents of  FIGS. 12 to 18 . 
     First, processing of the determination model MD2 in a learning stage will be described. 
     As illustrated in  FIG. 19 , the information processing apparatus  1  inputs, to the first neural network NN 1 , a pair of image data IM 1  and character string data ST 1  that are included in the first training data DT 1  and pairs of image data IM 1  and character string data ST 1  that are included in the second training data DT 2 . For example, as illustrated in  FIG. 19 , the information processing apparatus  1  inputs, to the first neural network NN 1 , a plurality of pairs including a pair of image data IM 1   a  and character string data ST 1   a  that are included in the first training data DT 1 , a pair of image data IM 1   b  and character string data ST 1   b  that are included in the second training data DT 2 , and a pair of image data IM 1   c  and character string data ST 1   c  that are included in the second training data DT 2 . 
     Then, for example, in response to reception of input of the pair of image data IM 1   a  and character string data ST 1   a , the first neural network NN 1  calculates and outputs an image vector indicating characteristics of the image data IM 1   a  and a character string vector indicating characteristics of the character string data ST 1   a . Similarly, for example, in response to reception of input of the pair of image data IM 1   b  and character string data ST 1   b , the first neural network NN 1  calculates and outputs an image vector indicating characteristics of the image data IM 1   b  and a character string vector indicating characteristics of the character string data ST 1   b . Moreover, for example, in response to reception of input of the pair of image data IM 1   c  and character string data ST 1   c , the first neural network NN 1  calculates and outputs an image vector indicating characteristics of the image data IM 1   c  and a character string vector indicating characteristics of the character string data ST 1   c.    
     Subsequently, the information processing apparatus  1  calculates, for example, an inner product of the image vector corresponding to the image data IM 1   a  and the character string vector corresponding to the character string data ST 1   a  as a degree of similarity between the image data IM 1   a  and the character string data ST 1   a . Similarly, the information processing apparatus  1  calculates, for example, an inner product of the image vector corresponding to the image data IM 1   b  and the character string vector corresponding to the character string data ST 1   b  as a degree of similarity between the image data IM 1   b  and the character string data ST 1   b . Moreover, the information processing apparatus  1  calculates, for example, an inner product of the image vector corresponding to the image data IM 1   c  and the character string vector corresponding to the character string data ST 1   c  as a degree of similarity between the image data IM 1   c  and the character string data ST 1   c.    
     Thereafter, for example, the information processing apparatus  1  adjusts the weight of the first neural network NN 1  such that a classification error calculated from the degree of similarity for each of the first training data DT 1  and the second training data DT 2  and information indicating the first training data DT 1  (correct pair) becomes small. 
     Furthermore, the information processing apparatus  1  calculates, for example, an element product of the image vector corresponding to the image data IM 1   a  and the character string vector corresponding to the character string data ST 1   a  and an element product of the image vector corresponding to the image data IM 1   b  and the character string vector corresponding to the character string data ST 1   b.    
     Then, the information processing apparatus  1  inputs, to the second neural network NN 2 , the element product corresponding to the image data IM 1   a  and the character string data ST 1   a  and the element product corresponding to the image data IM 1   b  and the character string data ST 1   b.    
     Subsequently, for example, in response to reception of input of the element product corresponding to the image data IM 1   a  and the character string data ST 1   a , the second neural network NN 2  calculates and outputs a degree of matching between the image data IM 1   a  and the character string data ST 1   a . Furthermore, for example, in response to reception of input of the element product corresponding to the image data IM 1   b  and the character string data ST 1   b , the second neural network NN 2  calculates and outputs a degree of matching between the image data IM 1   b  and the character string data ST 1   b.    
     Moreover, the information processing apparatus  1  adjusts the weights of the first neural network NN 1  and the second neural network NN 2  such that an error between the degree of matching between the image data IM 1   a  and the character string data ST 1   a  and a value corresponding to the first training data DT 1  (correct data) becomes small. Furthermore, the information processing apparatus  1  adjusts the weights of the first neural network NN 1  and the second neural network NN 2  such that an error between the degree of matching between the image data IM 1   b  and the character string data ST 1   b  and a value corresponding to the second training data DT 2  becomes small. 
     Next, processing of the determination model MD2 in an inference stage will be described. 
     As illustrated in  FIG. 20 , the information processing apparatus  1  inputs, to the first neural network NN 1 , a pair of image data IM 2  and character string data ST 2  that are included in the new training data DT 3 . 
     Then, for example, in response to reception of input of the pair of image data IM 2  and character string data ST 2 , the first neural network NN 1  calculates and outputs an image vector indicating characteristics of the image data IM 2  and a character string vector indicating characteristics of the character string data ST 2 . 
     Thereafter, the information processing apparatus  1  calculates, for example, an inner product of the image vector corresponding to the image data IM 2  and the character string vector corresponding to the character string data ST 2  as a degree of similarity between the image data IM 2  and the character string data ST 2 . 
     Then, the information processing apparatus  1  outputs, for example, the calculated degree of similarity to the operation terminal  2  as a degree of similarity between the image data IM 2  and the character string data ST 2  that are included in the new training data DT 3 . 
     Furthermore, the information processing apparatus  1  calculates, for example, an element product of the image vector corresponding to the image data IM 2  and the character string vector corresponding to the character string data ST 2 . Moreover, the information processing apparatus  1  inputs the calculated element product to the second neural network NN 2 . 
     Then, for example, in response to reception of input of the element product corresponding to the image data IM 2  and the character string data ST 2 , the second neural network NN 2  calculates and outputs a degree of matching between the image data IM 2  and the character string data ST 2 . 
     Thereafter, the information processing apparatus  1  outputs, for example, the degree of matching output from the second neural network NN 2  to the operation terminal  2  as a degree of matching between the image data IM 2  and the character string data ST 2  that are included in the new training data DT 3 . 
     As described above, the information processing apparatus  1  in the present embodiment generates, on the basis of the first training data DT 1  in which the image data IM 1  and the character string data ST 1  corresponding to the image data IM 1  are associated with each other, the second training data DT 2  by replacing one of the image data IM 1  and the character string data ST 1  that are included in the first training data DT 1  with another data. 
     Then, the information processing apparatus  1  generates, by using the first training data DT 1  and the second training data DT 2  as input data, a determination model that outputs information indicating which piece of training data in the first training data DT 1  and the second training data DT 2  is training data in which correspondence between the image data IM 1  and the character string data ST 1  is correct. 
     For example, in a case where a plurality of pairs of image data IM 1  and character string data ST 1  to be input to the determination model is generated, the information processing apparatus  1  in the present embodiment combines different character string data ST 1  with the same image data IM 1 . Furthermore, in this case, the information processing apparatus  1  combines different image data IM 1  with the same character string data ST 1 . Then, the information processing apparatus  1  trains the determination model by using the generated plurality of pairs. 
     With this configuration, the information processing apparatus  1  may cause the determination model to learn a detailed relationship between pieces of the training data. Therefore, the information processing apparatus  1  may generate the determination model that accurately determines correspondence between the image data IM 1  and the character string data ST 1 . 
     Note that, in the examples described above, a case has been described where the first neural network NN 1  that generates image vectors from the image data IM 1  and the image data IM 2  and the first neural network NN 1  that generates character string vectors from the character string data ST 1  and the character string data ST 2  are the same neural network, but these may be different neural networks from each other. 
     With this configuration, the information processing apparatus  1  may train the first neural network NN 1  that generates the image vectors by using only the image data IM 1 , and may also train the first neural network NN 1  that generates the character string vectors by using only the character string data ST 1 . Therefore, in this case, the information processing apparatus  1  does not need to train the first neural network NN 1  by inputting the image data IM 1  and the character string data ST 1  at the same time, and may efficiently generate the first neural network NN 1 . 
     All examples and conditional language provided herein are intended for the pedagogical purposes of aiding the reader in understanding the invention and the concepts contributed by the inventor to further the art, and are not to be construed as limitations to such specifically recited examples and conditions, nor does the organization of such examples in the specification relate to a showing of the superiority and inferiority of the invention. Although one or more embodiments of the present invention have been described in detail, it should be understood that the various changes, substitutions, and alterations could be made hereto without departing from the spirit and scope of the invention.