Patent Publication Number: US-2021191613-A1

Title: Ink data generation apparatus, method, and program

Description:
BACKGROUND 
     Technical Field 
     The present invention relates to an ink data generation apparatus, an ink data generation method, and an ink data generation program that generate ink data. 
     Description of the Related Art 
     Patent Document 1 discloses a method of generating a metadata block, in which N types of metadata identifying an input device are associated with M items of stroke data, and writing the metadata block into an ink file together with the stroke data. 
     PRIOR ART DOCUMENT 
     Patent Document 
     Patent Document 1: WO 2016/157400 
     BRIEF SUMMARY 
     Technical Problem 
     An object of the present invention is to provide an ink data generation apparatus, an ink data generation method, and an ink data generation program that are capable of improving ease of handling ink data when or after metadata is generated. 
     Technical Solution 
     An ink data generation apparatus according to a first aspect of the present invention is an apparatus that generates ink data including metadata describing meta-information regarding each of sets of strokes, the ink data generation apparatus including: a first acquisition section configured to acquire first set data representing stroke elements belonging to a first set to which the meta-information has not been assigned yet; a second acquisition section configured to acquire second set data representing stroke elements belonging to a second set to which the meta-information has already been assigned on a per stroke set basis; a set determination section configured to perform a determination determine, by comparing the stroke elements of the first set data and the stroke elements of the second set data using the first set data acquired by the first acquisition section and the second set data acquired by the second acquisition section, as to an inclusion relation between the first set and the second set; and a data generation section configured to generate first metadata for the first set described in a form that varies in accordance with a result of the determination performed by the set determination section. 
     An ink data generation apparatus according to a second aspect of the present invention is an apparatus that generates ink data including metadata describing meta-information regarding each of sets of strokes, the ink data generation apparatus including: a data acquisition section configured to acquire set data representing stroke elements belonging to a set of strokes on a set-by-set basis; and a data output section configured to output metadata indicating an inclusion relation between a plurality of sets on the basis of the set data acquired by the data acquisition section on a set-by-set basis. 
     An ink data generation method according to a third aspect of the present invention is a method for generating ink data including metadata describing meta-information regarding each of sets of strokes, the ink data generation method being implemented by one or a plurality of computers performing: a first acquisition step of acquiring first set data representing stroke elements belonging to a first set to which the meta-information has not been assigned yet; a second acquisition step of acquiring second set data representing stroke elements belonging to a second set to which the meta-information has already been assigned on a per stroke set basis; a determination step of performing a determination, by comparing the stroke elements of the first set data and the stroke elements of the second set data using the acquired first set data and the acquired second set data, as to an inclusion relation between the first set and the second set; and a generation step of generating first metadata for the first set described in a form that varies in accordance with a result of the determination at the determination step. 
     An ink data generation program according to a fourth aspect of the present invention is a program for generating ink data including metadata describing meta-information regarding each of sets of strokes, the ink data generation program causing one or a plurality of computers to perform: a first acquisition step of acquiring first set data representing stroke elements belonging to a first set to which the meta-information has not been assigned yet; a second acquisition step of acquiring second set data representing stroke elements belonging to a second set to which the meta-information has already been assigned on a per stroke set basis; a determination step of performing a determination, by comparing the stroke elements of the first set data and the stroke elements of the second set data using the acquired first set data and the acquired second set data, as to an inclusion relation between the first set and the second set; and a generation step of generating first metadata for the first set described in a form that varies in accordance with a result of the determination at the determination step. 
     Advantageous Effect 
     The present invention is capable of improving ease of handling ink data when or after metadata is generated. 
    
    
     
       BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS 
         FIG. 1  is an overall configuration diagram of an ink data generation apparatus according to an embodiment of the present invention. 
         FIG. 2  is a diagram illustrating an example of the ink data in  FIG. 1 . 
         FIG. 3  is a diagram illustrating content visualized on the basis of the ink data in  FIG. 2 . 
         FIG. 4  is a first flowchart for describing an operation of the ink data generation apparatus illustrated in  FIG. 1 . 
         FIG. 5  is a diagram illustrating a method of specifying a new set with a lasso operation. 
         FIG. 6A  is a diagram illustrating a method of selecting a subgroup. 
         FIG. 6B  is a diagram illustrating a method of extracting feature amounts in the subgroup. 
         FIG. 7A  is a diagram illustrating an example configuration of a discriminator. 
         FIG. 7B  is a diagram illustrating a method of deriving a histogram. 
         FIG. 8  is a diagram illustrating an example of a screen displayed in step S 5  in  FIG. 4 . 
         FIG. 9  is a diagram illustrating the positional relationships between the content and two lassos. 
         FIG. 10  is a diagram illustrating an example description of the ink data corresponding to  FIG. 9 . 
         FIG. 11  is a second flowchart for describing an operation of the ink data generation apparatus illustrated in  FIG. 1 . 
         FIG. 12  is a diagram illustrating the positional relationships between the content and three lassos. 
         FIG. 13  is a diagram illustrating an example description of the ink data corresponding to  FIG. 12 . 
         FIG. 14  is a diagram illustrating the positional relationships between the content and three lassos. 
         FIG. 15  is a diagram illustrating a first example description of the ink data corresponding to  FIG. 14 . 
         FIG. 16  is a diagram illustrating a second example description of the ink data corresponding to  FIG. 14 . 
         FIG. 17  is a diagram illustrating a third example description of the ink data corresponding to  FIG. 14 . 
         FIG. 18  is a diagram illustrating the positional relationships between the content and three lassos. 
         FIG. 19  is a diagram illustrating an example description of the ink data corresponding to  FIG. 18 . 
         FIG. 20  is a diagram illustrating results of assigning semantics attributes. 
     
    
    
     DETAILED DESCRIPTION 
     Structure of Ink Data Generation Apparatus  10   
     Overall Structure 
       FIG. 1  is an overall configuration diagram of an ink data generation apparatus  10  according to an embodiment of the present invention. The ink data generation apparatus  10  is an electronic device provided with a touchscreen display  12 , and is formed by, for example, a tablet terminal, a smart phone, or a personal computer. The ink data generation apparatus  10  specifically includes the touchscreen display  12 , a display driver IC (Integrated Circuit)  14 , a touch sensor IC  16 , a host processor  18 , and a memory  20 . 
     The touchscreen display  12  includes a display panel  22  capable of displaying visible content, and a touch sensor  24  arranged on the display panel  22 . The display panel  22  is capable of displaying a black-and-white image or a color image, and may be, for example, a liquid crystal panel or an organic EL (Electro-Luminescence) panel. The touch sensor  24  is provided with a plurality of X-line electrodes for sensing positions along an X-axis of a sensor coordinate system, and a plurality of Y-line electrodes for sensing positions along a Y-axis thereof. 
     The display driver IC  14  is an integrated circuit for performing drive control on the display panel  22 . The display driver IC  14  drives the display panel  22  on the basis of a display signal supplied from the host processor  18 . Content represented by ink data  50  is thus displayed on the display panel  22 . 
     The touch sensor IC  16  is an integrated circuit for performing drive control on the touch sensor  24 . The touch sensor IC  16  drives the touch sensor  24  on the basis of a control signal supplied from the host processor  18 . The touch sensor IC  16  thus implements “pen detection functions” that detect the state of a stylus  26 , and “touch detection functions” that detect a touch made by a finger of a user or the like. 
     Examples of the pen detection functions include a function of scanning the touch sensor  24 , a function of receiving and analyzing a downlink signal, a function of estimating the state (e.g., position, posture, pen pressure, etc.) of the stylus  26 , and a function of generating and transmitting an uplink signal including a command for the stylus  26 . Meanwhile, examples of the touch detection functions include a function of two-dimensionally scanning the touch sensor  24 , a function of acquiring a detection map on the touch sensor  24 , and a function of classifying regions (e.g., classifying a finger, a palm, etc.) on a detection map. 
     Thus, a user interface (hereinafter referred to as a UI section  28 ) is formed by combining input functions implemented by the touch sensor  24  and the stylus  26  and an output function implemented by the display panel  22 . 
     The host processor  18  is formed by a processing/computing device, examples of which include a CPU (Central Processing Unit), an MPU (Micro-Processing Unit), and a GPU (Graphics Processing Unit). The host processor  18  reads a program stored in the memory  20  and executes the program, and thereby functions as a first acquisition section  30 , a second acquisition section  32 , an attribute estimation section  34 , a set determination section  36 , a data generation section  38  (a data output section), and a drawing processing section  40 . 
     The memory  20  is formed by a non-temporary computer-readable storage medium. Here, examples of such a computer-readable storage medium include storage devices including a hard disk drive (HDD) and a solid-state drive (SSD), and portable media including a magneto-optical disk, a ROM (Read-Only Memory), a CD-ROM (Compact Disc-Read-Only Memory), and a flash memory. In the example of the present drawing, new set data  42  (i.e., first set data), existing set data  44  (i.e., second set data), a learning parameter group  46 , and the ink data  50  are stored in the memory  20 . 
     Example of Ink Data  50   
       FIG. 2  is a diagram illustrating an example of the ink data  50  in  FIG. 1 . The following description will be made taking an example of InkML (Ink Markup Language) which describes a digital ink in an XML (eXtensible Markup Language) format. Note that the data format, i.e., what is called “ink markup language,” of the ink data  50  is not limited to InkML, and that WILL (Wacom Ink Layer Language) or ISF (Ink Serialized Format), for example, may alternatively be used. Further, describing the ink data  50  using a data structure format of JSON (JavaScript (registered trademark) Object Notation) facilitates exchange of the data between various types of software and programming languages. 
     In the example of the present drawing, the ink data  50  includes stroke data  51  that describes at least one stroke (in the example of the present drawing, 24 strokes). As will be understood from the present drawing, each stroke is described by a plurality of items of point data sequentially arranged in &lt;trace&gt; tags. The items of point data are each made up of at least an indicated position (X-coordinate and Y-coordinate), and are separated by a delimiter, such as a comma. For the sake of convenience in illustration, only items of point data that represent a starting point and an ending point of each stroke are depicted, while items of point data that represent a plurality of intermediate points are omitted. Note that the items of point data may include, in addition to the aforementioned indicated positions, an order of writing, a pen pressure and/or a posture of the stylus  26 , and/or the like. 
       FIG. 3  is a diagram illustrating content visualized on the basis of the ink data  50  in  FIG. 2 . On the display panel  22  ( FIG. 1 ), a two-dimensional coordinate system (hereinafter referred to as a sensor coordinate system: X-Y) for identifying positions detected on the touch sensor  24  is defined. This content includes a character string “This is a pencil.” and a drawing of a pencil. It should be noted that numbers in parentheses indicate the writing order of strokes (stroke IDs (IDentifications)=01-24), and do not constitute a part of the content. 
     First Operation of Ink Data Generation Apparatus  10   
     The ink data generation apparatus  10  according to this embodiment has the structure described above. First, a first operation of the ink data generation apparatus  10  will be described below with reference to a flowchart of  FIG. 4  and  FIGS. 5 to 10 . This “first operation” means an operation of assisting a user in assigning meta-information (e.g., a data attribute, a creator or an author, a date and time of creation, a used device, etc.). 
     Prior to the first operation, the drawing processing section  40  analyzes the ink data  50  read from the memory  20 , performs a desired rasterization process on the stroke data  51 , and generates a display signal representing the content. The display driver IC  14  drives the display panel  22  on the basis of the display signal supplied from the host processor  18 . The visualized content is thus displayed on the display panel  22 . 
     Description of First Operation 
     At step S 1  in  FIG. 4 , the host processor  18  determines whether or not an operation of specifying a new set (i.e., a first set) of strokes to which meta-information is to be assigned has been accepted. This “specifying operation” is an operation for selecting one or more strokes from among a plurality of strokes, and examples thereof include a “lasso operation” of enclosing a selection target with a lasso. 
       FIG. 5  is a diagram illustrating a method of specifying a new set with the lasso operation. The user performs a writing operation of enclosing the whole of strokes to which meta-information is to be assigned while moving the stylus  26  on a touch surface on the display panel  22 . This handwriting input causes a lasso Ltmp to be displayed on the visualized content in an overlapping manner. 
     If a specifying operation by the user has not been accepted yet (step S 1 : NO), control stays at step S 1  until this specifying operation is accepted. Meanwhile, if a specifying operation by the user has been accepted (step S 1 : YES), control proceeds to the next step S 2 . 
     At step S 2 , the first acquisition section  30  acquires data (hereinafter referred to as the new set data  42 ) representing stroke elements that belong to the new set on the basis of the content of the specifying operation accepted at step S 1 . The new set data  42  is data capable of identifying the stroke elements, and may be, for example, stroke IDs, which are identification information for the strokes. In the example of  FIGS. 2 and 5 , the first acquisition section  30  acquires stroke IDs=01-18 as the new set data  42 . 
     At step S 3 , the host processor  18  determines whether a mode (hereinafter referred to simply as an “input support mode”) for providing assistance in inputting the meta-information is ON or OFF. If this input support mode is “OFF” (step S 3 : OFF), control proceeds to step S 5 , omitting a performance of step S 4 . Meanwhile, if the input support mode is “ON” (step S 3 : ON), control proceeds to the next step S 4 . 
     At step S 4 , the attribute estimation section  34  estimates the attribute to be assigned to the new set on the basis of mutual relations between the positions or shapes of the plurality of strokes that form the new set specified at step S 1 . Here, the attribute estimation section  34  estimates a semantics attribute of the new set using a discriminator  70  (see  FIG. 7A ) constructed by machine learning. A method of estimating the attribute will now be described in detail below with reference to  FIGS. 6A to 7B . 
       FIGS. 6A and 6B  are diagrams illustrating an example method of generating input data for the discriminator  70 . In more detail,  FIG. 6A  is a diagram illustrating a method of selecting a subgroup  74 , and  FIG. 6B  is a diagram illustrating a method of extracting feature amounts in the subgroup  74 . 
     As illustrated in  FIG. 6A , the attribute estimation section  34  selects one subgroup  74  made up of a predetermined number of strokes from among a group  72  composed of 18 stroke elements (stroke IDs=01-18). It is assumed here that five strokes having five consecutive IDs=01-05 starting with ID=01 have been extracted. 
     As illustrated in  FIG. 6B , the attribute estimation section  34  sets a square region encompassing all the five strokes that belong to the subgroup  74 , and defines a two-dimensional coordinate system (hereinafter referred to as a normalized coordinate system; X′-Y′) that matches this region. An origin O′ of this normalized coordinate system corresponds to a vertex of the square region that is located closest to an origin O of the sensor coordinate system. An X′-axis of the normalized coordinate system is parallel to the X-axis of the sensor coordinate system, while a Y′-axis of the normalized coordinate system is parallel to the Y-axis of the sensor coordinate system. In addition, the scales of the X′-axis and the Y′-axis are normalized so that coordinates of four vertices defining the square region will be (0, 0), (1, 0), (0, 1), and (1, 1). 
     Referring to the stroke data  51  ( FIG. 2 ), the attribute estimation section  34  acquires coordinate values (X, Y) of a starting point Ps of each stroke and coordinate values (X, Y) of an ending point Pe of the stroke in the sensor coordinate system. Then, the attribute estimation section  34  derives coordinate values (X′, Y′) of the starting point Ps of the stroke and coordinate values (X′, Y′) of the ending point Pe of the stroke in the normalized coordinate system by performing linear transformation of the coordinate system. 
       FIGS. 7A and 7B  are diagrams illustrating an example method of estimating the semantics attribute. In more detail,  FIG. 7A  is a diagram illustrating an example configuration of the discriminator  70 , and  FIG. 7B  is a diagram illustrating a method of deriving a histogram. 
     As illustrated in  FIG. 7A , the discriminator  70  is formed by a hierarchical neural network made up of an input layer  76 , intermediate layers  78 , and an output layer  80 , for example. An algorithm of the discriminator  70  is determined by values of the learning parameter group  46 , which is a collection of learning parameters. The learning parameter group  46  may include, for example, a coefficient describing an activation function of a unit corresponding to a neuron, a weighting coefficient corresponding to strength of synaptic junction, the number of units constituting each layer, and the number of intermediate layers  78 . The learning parameter group  46  is stored in the memory  20  ( FIG. 1 ) with the values thereof determined by completion of learning, and is read at appropriate times as necessary. 
     The input layer  76  is a layer at which feature amounts concerning the starting and ending points of the strokes are inputted and is made up of 20 units in the example of the present drawing. These feature amounts form an input vector composed of 20 components, i.e., [1] the X′-coordinates of the starting points Ps, [2] the Y′-coordinates of the starting points Ps, [3] the X′-coordinates of the ending points Pe, and [4] the Y′-coordinates of the ending points Pe, arranged according to the order of the stroke ID. 
     The output layer  80  is a layer at which a collection of label values (hereinafter referred to as a label group) representing semantics attributes is outputted, and is made up of six units in the example of the present drawing. This label group forms an output vector composed of six components indicating the likelihoods of [1] Text (English), [2] Text (Japanese), [3] Drawing (Graphic), [4] Drawing (Illustration), [5] Numerical Equation, and [6] Chemical Formula. 
     The discriminator  70  accepts input of the feature amounts generated from the subgroup  74  via the input layer  76 , passes the feature amounts through the intermediate layers  78 , and outputs a label group corresponding to the subgroup  74  to the output layer  80 . In the case where each label is defined in the range [ 0 ,  1 ], for example, the semantics attribute that has the greatest label value, e.g., “Text (English),” is selected. 
     As illustrated in  FIG. 7B , after identifying the semantics attribute of the first subgroup  74 , the attribute estimation section  34  selects a next subgroup  74  made up of five strokes (IDs=02-06), with the stroke IDs incremented by one, and performs the above-described discrimination process thereon. Then, the attribute estimation section  34  totals results of discrimination using a plurality of subgroups  74 , and creates a histogram in which the “frequency” is the number of semantics attributes. In the example of the present drawing, it is estimated that “Text (English),” which has the greatest frequency in the histogram, is the semantics attribute to be assigned to the new set (i.e., the group  72  in  FIG. 6A ). 
     At step S 5  in  FIG. 4 , the host processor  18  causes an input window  82  for input of the meta-information (e.g., an annotation) to be displayed on the display panel  22 . The input window  82  is displayed, for example, at a position that does not overlap with a closed region enclosed by the lasso Ltmp. 
     As illustrated in  FIG. 8 , two user controls  84  and  86  and buttons  88  labeled as [OK] and [Cancel] are arranged in the input window  82 . The first user control  84  is, for example, a pull-down menu configured to enable input of the semantics attribute. The second user control  86  is, for example, a text box configured to enable input of a value attribute. 
     If the input support mode is “ON,” i.e., when control has passed through “step S 3 : ON” in  FIG. 4 , the semantics attribute obtained by the estimation at step S 4  is displayed as an initial value (i.e., a recommended value) in an input box of the user control  84 . Meanwhile, if the input support mode is “OFF,” i.e., when control has passed through “step S 3 : OFF” in  FIG. 4 , an indication (e.g., a blank space) of no attribute being selected is displayed in the input box of the user control  84 . 
     When a semantics attribute that corresponds to an intention of the user has been presented, the user can omit an operation on the user control  84 . Meanwhile, when a semantics attribute that does not correspond to the intention of the user has been presented, the user may operate the user control  84  so as to select the intended attribute. 
     At step S 6 , the host processor  18  determines whether or not an operation of confirming the data input, more specifically, an operation of touching the [OK] button  88  ( FIG. 8 ), has been accepted. If the confirming operation has not been accepted yet (step S 6 : NO), control stays at step S 6  until this confirming operation is accepted. Meanwhile, if the confirming operation has been accepted (step S 6 : YES), control proceeds to step S 7 , which will be described below with reference to  FIG. 11 . 
       FIG. 9  is a diagram illustrating the positional relationships between the content and two lassos Lprv and Ltmp. It is assumed that the user performs a first annotation operation for a stroke set enclosed by the lasso Lprv, and thereafter performs a second annotation operation for a stroke set enclosed by the lasso Ltmp. By the first annotation operation, meta-information in which the semantics attribute (Sem) is “None” and the value attribute (Value) is “Pencil” is assigned to the set including the stroke elements with IDs=11-17 and 19-23. By the second annotation operation, meta-information in which the semantics attribute is “Text (English)” and the value attribute is “None” is assigned to the set including the stroke elements with IDs=01-18. 
       FIG. 10  is a diagram illustrating an example description of the ink data  50  corresponding to  FIG. 9 . The ink data  50  is made up of the stroke data  51  and metadata  52 . The metadata  52  includes two meta-chunks  53  and  54 . The first meta-chunk  53  describes the meta-information assigned to the stroke set indicated by the lasso Lprv. The second meta-chunk  54  describes the meta-information assigned to the stroke set indicated by the lasso Ltmp. 
     Effects by First Operation 
     As described above, the ink data generation apparatus  10  is an apparatus that generates the ink data  50  including the metadata  52  describing the meta-information regarding each of sets of strokes, the apparatus including: the first acquisition section  30  that acquires the new set data  42  representing the stroke elements belonging to the new set to which the meta-information has not been assigned yet; the attribute estimation section  34  that estimates the attribute to be assigned to the new set on the basis of the mutual relations between the positions or shapes of the plurality of strokes that form the new set; and the data generation section  38  that generates the meta-chunk  53  and  54  (i.e., first metadata) for assigning the attribute obtained by the estimation to the new set. 
     Meanwhile, a corresponding ink data generation method and a corresponding program cause one or a plurality of processors (e.g., the host processor  18 ) to perform: a first acquisition step (S 2  in  FIG. 4 ) of acquiring the new set data  42  representing the stroke elements belonging to the new set to which the meta-information has not been assigned yet; an estimation step (S 4  in  FIG. 4 ) of estimating the attribute to be assigned to the new set on the basis of the mutual relations between the positions or shapes of the plurality of strokes that form the new set; and a generation step (S 10  or S 11  in  FIG. 11 , which will be described below) of generating the meta-chunk  53  and  54  (i.e., the first metadata) for assigning the attribute obtained by the estimation to the new set. 
     The above configuration makes it possible to automatically select an attribute that is appropriate with a high probability by taking into account the mutual relations between the positions or shapes of the plurality of strokes, leading to a reduced burden when the user inputs the attribute. 
     In addition, the attribute estimation section  34  may be configured to include the discriminator  70  that accepts the input of the feature amounts concerning the starting and ending points of the strokes and outputs the label group of the semantics attributes. An improvement in accuracy in the estimation of the semantic attribute can be achieved by focusing on correlations between the starting and ending points of the strokes and semantic attributes. 
     Moreover, the ink data generation apparatus  10  may further include the UI section  28  configured to be capable of displaying a plurality of strokes simultaneously in a display area and to enable the stroke elements belonging to the new set to be selected from among the plurality of strokes based on a handwriting input by the user. The UI section  28  may further display the user control  84 , which enables the input of the semantics attribute to be assigned to the new set, and the data generation section  38  may generate the meta-chunks  53  and  54  indicating the semantics attribute inputted by the user control  84 . This enables smooth input of the semantics attribute using the user interface. 
     Furthermore, the UI section  28  may display the semantic attribute obtained by the estimation by the attribute estimation section  34  as the initial value of the user control  84 . This will increase the probability that the need to perform an operation of changing the state of the user control  84  from an initial state thereof will be eliminated, while affording the user an opportunity to make a change thereto. 
     Second Operation of Ink Data Generation Apparatus  10   
     Next, a second operation of the ink data generation apparatus  10  will be described below with reference to a flowchart of  FIG. 11  and  FIGS. 12 to 17 . This “second operation” means an operation of generating the ink data  50  on the basis of the meta-information inputted by the first operation. Here, the data generation section  38  performs “flow-type” data processing in which the ink data  50  is updated as necessary each time the meta-information is assigned. 
     Description of Second Operation 
     At step S 7  in  FIG. 11 , the second acquisition section  32  acquires data (hereinafter referred to as the existing set data  44 ) representing stroke elements that belong to a set (hereinafter referred to as an existing set) of strokes to which the meta-information has already been assigned. Similarly to the new set data  42 , the existing set data  44  is data capable of identifying the stroke elements, and may be, for example, stroke IDs, which are identification information for the strokes. 
     At step S 8 , using the new set data  42  acquired at step S 2  in  FIG. 4  and the existing set data  44  acquired at step S 7 , the set determination section  36  performs a determination as to an inclusion relation between the new set and the existing set by comparing the stroke elements of the new set data  42  and the stroke elements of the existing set data  44 . Specifically, the set determination section  36  determines which of the following is true: (1) the existing set is a superset of the new set; (2) the existing set is a subset of the new set; and (3) there is not an inclusion relation therebetween. It should be noted that, in the case where there are a plurality of existing sets, the set determination section  36  performs the determination as to the inclusion relation with respect to each of the existing sets. 
     At step S 9 , the set determination section  36  determines whether or not there is an existing set that has an inclusion relation with the new set. 
       FIG. 12  is a diagram illustrating the positional relationships between the content and three lassos Lpr 1 , Lpr 2 , and Ltmp. It is assumed here that the user performs a ninth annotation operation for a stroke set enclosed by the lasso Ltmp after performing the first to eighth annotation operations. By the ninth annotation operation, meta-information in which the semantics attribute (Sem) is “Drawing” and the value attribute (Value) is “None” is assigned to the set including the stroke elements with IDs=19-24. For the sake of convenience in illustration, depiction of third to eighth lassos is omitted. 
     As will be understood from the present drawing, an existing set specified by the lasso Lpr 1  (i.e., the first annotation operation) includes the stroke elements with IDs=19-23, but does not include the stroke element with ID=24. Meanwhile, an existing set specified by the lasso Lpr 2  (i.e., the second annotation operation) does not include the stroke elements with IDs=19-24. In this case, it is determined that there is not an inclusion relation between the new set and any existing set (step S 9 : NO), and control proceeds to step S 10 . 
     At step S 10 , the data generation section  38  generates a meta-chunk  55  describing the meta-information, the input of which has been confirmed at step S 6  in  FIG. 4 , and places the meta-chunk  55  at a position according to a predetermined order of meta-information assignment. 
       FIG. 13  is a diagram illustrating an example description of the ink data  50  corresponding to  FIG. 12 . The ink data  50  is made up of the stroke data  51  and metadata  52 A. The metadata  52 A is made up of nine meta-chunks including the meta-chunks  53 ,  54 , and  55 . The first meta-chunk  53  describes the meta-information assigned to the existing set specified by the lasso Lpr 1 . The second meta-chunk  54  describes the meta-information assigned to the existing set specified by the lasso Lpr 2 . The ninth meta-chunk  55  describes the meta-information assigned to the new set specified by the lasso Ltmp. For the sake of convenience in illustration, illustration of the third to eighth meta-chunks is omitted. 
     In the example of the present drawing, a rule stipulating that meta-information newly assigned should be added at a lower position one item after another is provided, and accordingly, the ninth meta-chunk  55  is added at a bottom of the metadata  52 . In the case where, in contrast to the example of the present drawing, a rule stipulating that meta-information newly assigned should be added at an upper position one item after another is provided, the meta-chunk  55  is added at a top of the metadata  52 . 
     Meanwhile,  FIG. 14  is a diagram illustrating the positional relationships between the content and three lassos Lpr 1 , Lpr 2 , and Ltmp.  FIG. 14  is different from  FIG. 12  only in the region specified by the lasso Ltmp. By the ninth annotation operation, meta-information in which the semantics attribute is “Drawing” and the value attribute is “None” is assigned to a set including the stroke elements with IDs=19-23. 
     As will be understood from the present drawing, the existing set specified by the lasso Lpr 1  includes all the stroke elements (IDs=19-23), and therefore corresponds to a “superset” of the new set. Accordingly, it is determined that there is an inclusion relation between the new set and the existing set (step S 9 : YES), and control proceeds to step S 11 . 
     At step S 11 , the data generation section  38  generates a meta-chunk  56 ,  57 , or  58  (i.e., the first metadata) so as to be dependent on the meta-chunk  54  (i.e., second metadata) of the existing set which is a superset or subset of the new set. A specific example of a method of generating the meta-chunk  56 ,  57 , or  58  will now be described below with reference to  FIGS. 15 to 17 . 
       FIG. 15  is a diagram illustrating a first example description of the ink data  50  corresponding to  FIG. 14 . The ink data  50  is made up of the stroke data  51  and metadata  52 B. The metadata  52 B is made up of nine meta-chunks including the meta-chunks  53 ,  54 , and  56 . Description of the meta-chunks  53  and  54 , which are not changed from those in  FIG. 13 , is omitted. 
     The ninth meta-chunk  56  describes the meta-information assigned to the new set specified by the lasso Ltmp. In the example of the present drawing, the ninth meta-chunk  56  is added so as to be incorporated in the meta-chunk  53  despite the rule stipulating that meta-information newly assigned should be added at a lower position one item after another. 
     As described above, in the case where it has been determined that there is an inclusion relation between the new set and any existing set, the data generation section  38  may generate the meta-chunk  56  for the new set at a position within the meta-chunk  53  for that existing set. This example description enables the meta-chunks  53  and  56  having an inclusion relation therebetween to be associated with each other when generating the ink data  50 . 
       FIG. 16  is a diagram illustrating a second example description of the ink data  50  corresponding to  FIG. 14 . The ink data  50  is made up of the stroke data  51  and metadata  52 C. The metadata  52 C is made up of nine meta-chunks including the meta-chunks  53 ,  54 , and  57 . Description of the meta-chunks  53  and  54 , which are not changed from those in  FIG. 13 , is omitted. 
     The ninth meta-chunk  57  describes the meta-information assigned to the new set specified by the lasso Ltmp. In the example of the present drawing, the ninth meta-chunk  56  is added at a position immediately below the meta-chunk  53  (immediately above the meta-chunk  54 ) despite the rule stipulating that meta-information newly assigned should be added at a lower position one item after another. 
     As described above, in the case where it has been determined that there is an inclusion relation between the new set and any existing set, the data generation section  38  may generate the meta-chunk  57  for the new set at a position immediately above or immediately below the meta-chunk  53  for that existing set. This example description enables the meta-chunks  53  and  57  having an inclusion relation therebetween to be associated with each other when generating the ink data  50 . 
       FIG. 17  is a diagram illustrating a third example description of the ink data  50  corresponding to  FIG. 14 . The ink data  50  is made up of the stroke data  51  and metadata  52 D. The metadata  52 D is made up of nine meta-chunks including the meta-chunks  53 ,  54 , and  58 . Description of the meta-chunks  53  and  54 , which are not changed from those in  FIG. 13 , is omitted. 
     The ninth meta-chunk  58  describes the meta-information assigned to the new set specified by the lasso Ltmp. In the example of the present drawing, the ninth meta-chunk  58  is added at a bottom of the metadata  52 D in accordance with the rule stipulating that meta-information newly assigned should be added at a lower position one item after another. However, the meta-chunk  58  is described in a form having a reference to an identification code (here, group ID=g1) of the stroke set included in the meta-chunk  53  in &lt;traceView&gt; tags. 
     As described above, in the case where it has been determined that there is an inclusion relation between the new set and any existing set, the data generation section  38  may generate the meta-chunk  58  for the new set in a form that refers to the identification code of the existing set included in the meta-chunk  53 . This example description enables the meta-chunks  53  and  58  having an inclusion relation therebetween to be associated with each other when generating the ink data  50 . 
     After the data generation section  38  generates the meta-chunk  55 ,  56 ,  57 , or  58  described in a form that varies in accordance with a result of the determination at step S 8  as described above (step S 10  or S 11 ), control proceeds to the next step S 12 . 
     At step S 12  in  FIG. 11 , the data generation section  38  outputs the ink data  50  generated/updated at step S 10  or S 11 . Specific examples of this “output” include storage into the memory  20 , writing into a data file, and transmitting to an external device. The operation of the ink data generation apparatus  10  is thus completed. 
     Effects by Second Operation 
     As described above, the ink data generation apparatus  10  is an apparatus that generates the ink data  50  including the metadata  52  describing the meta-information regarding each of sets of strokes, the apparatus including: the first acquisition section  30  that acquires the new set data  42  (i.e., the first set data) representing the stroke elements belonging to the new set (i.e., the first set) to which the meta-information has not been assigned yet; the second acquisition section  32  that acquires the existing set data  44  (i.e., the second set data) representing the stroke elements belonging to the existing set (i.e., a second set) to which the meta-information has already been assigned on a per stroke set basis; the set determination section  36  that performs a determination as to the inclusion relation between the new set and the existing set by comparing the stroke elements of the new set data  42  and the stroke elements of the existing set data  44  using the acquired new set data  42  and the acquired existing set data  44 ; and the data generation section  38  that generates the first metadata (i.e., the meta-chunk  55 ,  56 ,  57 , or  58 ) for the new set described in a form that varies in accordance with the result of the determination performed by the set determination section  36 . 
     Meanwhile, a corresponding ink data generation method and a corresponding program cause one or a plurality of processors (e.g., the host processor  18 ) to perform: the first acquisition step (S 2  in  FIG. 4 ) of acquiring the new set data  42  representing the stroke elements belonging to the new set to which the meta-information has not been assigned yet; a second acquisition step (S 7 ) of acquiring the existing set data  44  representing the stroke elements belonging to the existing set to which the meta-information has already been assigned on a per stroke set basis; a determination step (S 8 ) of performing a determination as to the inclusion relation between the new set and the existing set by comparing the stroke elements of the new set data  42  and the stroke elements of the existing set data  44  using the acquired new set data  42  and the acquired existing set data  44 ; and a generation step (S 10  or S 11 ) of generating the meta-chunk  55 ,  56 ,  57 , or  58  described in a form that varies in accordance with the result of the determination at the determination step. 
     The above configuration enables recognizing the inclusion relation between the sets of strokes by analyzing the description form of the metadata  52 , which makes it possible to perform various processes that take into account the recognized inclusion relation, such as editing support (e.g., cross-referencing of meta-information), automatic editing (e.g., combining/substitution/inheritance of meta-information), etc., on the meta-information. This leads to an improved ease of handling the ink data  50  when or after the metadata  52  is generated. 
     In addition, when it has been determined that there is an inclusion relation between the existing set and the new set, the data generation section  38  may generate the meta-chunk  56 ,  57 , or  58  so as to be dependent on the meta-chunk  53  for the existing set which is a superset or a subset. The inclusion relation therebetween is made easier to recognize by configuring a dependence relationship between the meta-chunks  53  and  56 - 58 . 
     Modification 
     An example of generating ink data  60  for which WILL is used as an ink markup language in accordance with a notation of JSON will now be described below. 
     Example of Generation of Ink Data  60   
       FIG. 18  is a diagram illustrating the positional relationships between the content and three lassos Lpr 1 , Lpr 2 , and Lpr 3 . It is assumed that the user performs an eleventh annotation operation for a stroke set enclosed by the lasso Lpr 1 , and performs a twelfth annotation operation for a stroke set enclosed by the lasso Lpr 2 . By the eleventh annotation operation, meta-information in which the type (Type) is “Word” and the value (Value) is “This” is assigned to the set including the stroke elements with IDs=01-06. By the twelfth annotation operation, meta-information in which the type is “Word” and the value is “is” is assigned to the set including the stroke elements with IDs=07-09. 
     It is assumed that, after performing the eleventh to eighteenth annotation operations, the user performs a nineteenth annotation operation for a stroke set enclosed by the lasso Lpr 3 . By the nineteenth annotation operation, meta-information in which the type is “Text Line” and the value is “None” is assigned to the set including the stroke elements with IDs=01-09. For the sake of convenience in illustration, depiction of thirteenth to eighteenth lassos is omitted. 
     Here, the data generation section  38  performs “stock-type” data processing in which the ink data  60  is collectively updated at a predetermined timing after accumulating items of meta-information assigned by the user. In this case, the data generation section  38  acquires the result of the determination performed by the set determination section  36 , and, when the above-described inclusion relation is satisfied, generates the ink data  60  so as to include a meta-chunk  65  describing relevant meta-information in a hierarchical manner. In addition, the data generation section  38  may determine meta-information that has not been assigned yet from a relation with meta-information already assigned to a superset or a subset, and automatically add a code statement  68  for assigning the determined meta-information. 
       FIG. 19  is a diagram illustrating an example description of the ink data  60  corresponding to  FIG. 18 . The ink data  60  includes stroke data  61  describing strokes and stroke sets, and metadata  62  describing semantics attributes. The metadata  62  is made up of nine meta-chunks including meta-chunks  63 ,  64 , and  65 . The eleventh meta-chunk  63  describes the meta-information assigned to the set (tg 11 ) specified by the lasso Lpr 1 . The twelfth meta-chunk  64  describes the meta-information assigned to the set (tg 12 ) specified by the lasso Lpr 2 . 
     The nineteenth meta-chunk  65  describes the meta-information assigned to the set (tg 19 ) specified by the lasso Lpr 3 . In the example of the present drawing, the nineteenth meta-chunk  65  is added at a bottom of the metadata  62  in accordance with the rule stipulating that meta-information newly assigned should be added at a lower position one item after another. However, hierarchizing data  66  that defines the two sets (tg 11  and tg 12 ) as subsets using “has_subgroup” is added in the meta-chunk  65 . 
     In addition, a code statement  68  indicating that the value of the superset (tg 19 ) is “This is” is added in the meta-chunk  65 . This “This is” is a character string obtained by combining the value of the subset (tg 11 ), “This,” and the value of the subset (tg 12 ), “is.” 
     As a result, as illustrated in  FIG. 20 , to the set (tg 19 ) including the stroke elements with IDs=01-09, “Type: Text Line” is assigned as meta-information, and “Value: This is” and “Subgroup: tg 11 , tg 12 ” are automatically assigned as meta-information. In the example of the present drawing, the meta-information “This is” is automatically assigned when the value is “None,” but the meta-information does not have to be assigned depending on the intention of the user. 
     As described above, in the case where it has been determined that there is an inclusion relation between the first set (tg 19 ) and the second set (tg 11  or tg 12 ), the data generation section  38  may generate the meta-chunk  65  for the first set in a form that refers to the identification code of the second set (tg 11  or tg 12 ). This example description enables the meta-chunks  63  to  65  having an inclusion relation therebetween to be associated with one another when generating the ink data  60 . 
     Effects by Modification 
     As described above, the ink data generation apparatus  10  is an apparatus that generates the ink data  60  including the metadata  62  describing the meta-information regarding each of sets of strokes, the apparatus including: the data acquisition section (i.e., the first acquisition section  30  and the second acquisition section  32 ) that acquires the set data (i.e., the new set data  42  and the existing set data  44 ) representing stroke elements belonging to a set of strokes on a set-by-set basis; and the data output section (i.e., the data generation section  38 ) that outputs the ink data  60  including the metadata  62  indicating an inclusion relation between a plurality of sets on the basis of the set data acquired on a set-by-set basis. 
     Meanwhile, a corresponding ink data generation method and a corresponding program cause one or a plurality of processors (e.g., the host processor  18 ) to perform: an acquisition step (S 2  in  FIG. 4  and S 7  in  FIG. 11 ) of acquiring the set data representing stroke elements belonging to a set of strokes on a set-by-set basis; and an output step (S 12  in  FIG. 11 ) of outputting the ink data  60  including the metadata  62  indicating an inclusion relation between a plurality of sets on the basis of the set data acquired on a set-by-set basis. 
     The above configuration enables recognizing the inclusion relation between the sets of strokes by analyzing the content of the outputted metadata  62 , which makes it possible to perform various processes that take into account the recognized inclusion relation, such as editing support, automatic editing, etc., on the meta-information. This leads to an improved ease of handling the ink data  60  when or after the metadata  62  is outputted. 
     In addition, the data output section may output the metadata including the meta-chunk  65 , which describes a semantics attribute based on a data structure indicating the inclusion relation between the sets. 
     DESCRIPTION OF REFERENCE SYMBOLS 
       10 : Ink data generation apparatus,  12 : Touchscreen display,  18 : Host processor,  20 : Memory,  28 : UI section,  30 : First acquisition section,  32 : Second acquisition section,  34 : Attribute estimation section,  36 : Set determination section,  38 : Data generation section (data output section),  40 : Drawing processing section,  42 : New set data (first set data),  44 : Existing set data (second set data),  50 ,  60 : Ink data,  51 ,  61 : Stroke data,  52  (A/B/C/D),  62 : Metadata,  53 ,  54 ,  65 : Meta-chunk (second metadata),  55  to  58 ,  63 ,  64 : Meta-chunk (first metadata)