DISPLAY APPARATUS, INFORMATION PROCESSING METHOD, AND RECORDING MEDIUM

A display apparatus includes circuitry that receives a plurality of stroke data input to a touch panel by an input device and displays the plurality of stroke data. The plurality of stroke data includes first stroke data and second stroke data input after the first stroke data. The circuitry sets a determination area differently depending on whether an elapsed time from separation of the input device from the touch panel after input of the first stroke data exceeds a threshold. The determination area is for determining whether to include the second stroke data in a recognition group including the first stroke data. The circuitry performs character recognition on the recognition group and displays, on a screen, a result of the character recognition.

CROSS-REFERENCE TO RELATED APPLICATIONS

This patent application is based on and claims priority pursuant to 35 U.S.C. § 119(a) to Japanese Patent Application No. 2021-107458, filed on Jun. 29, 2021, in the Japan Patent Office, the entire disclosure of which is hereby incorporated by reference herein.

BACKGROUND

Technical Field

Embodiments of the present disclosure relate to a display apparatus, an information processing method, and a recording medium.

Related Art

There are display apparatuses that convert handwritten data to a character string (character codes) and displays the character string on a screen by using a handwriting recognition technology. A display apparatus having a relatively large touch panel is used in a conference room or the like and is shared by a plurality of users as an electronic whiteboard or the like. In some cases, a display apparatus is used as a written communication tool.

A related-art display apparatus imposes on a user a constraint that the user has to input handwriting in an input frame, and there has been proposed a technology for eliminating the input frame for converting the handwriting data.

SUMMARY

An embodiment provides a display apparatus includes circuitry that receives a plurality of stroke data input to a touch panel by an input device and displays the plurality of stroke data. The plurality of stroke data includes first stroke data and second stroke data input after the first stroke data. The circuitry sets a determination area differently depending on whether an elapsed time from separation of the input device from the touch panel after input of the first stroke data exceeds a threshold. The determination area is for determining whether to include the second stroke data in a recognition group including the first stroke data. The circuitry performs character recognition on the recognition group and displays, on a screen, a result of the character recognition.

Another embodiment provides an information processing method including receiving a plurality of stroke data input to a touch panel by an input device and displaying the plurality of stroke data. The plurality of stroke data includes first stroke data and second stroke data input after the first stroke data. The method further includes setting a determination area differently depending on whether an elapsed time from separation of the input device from the touch panel after input of the first stroke data exceeds a threshold. The determination area is for determining whether to include the second stroke data in a recognition group including the first stroke data. The method further includes performing character recognition of the recognition group and displaying, on a screen, a result of the character recognition.

Another embodiment provides a non-transitory recording medium storing a plurality of program codes which, when executed by one or more processors, causes the processors to perform the method described above.

DETAILED DESCRIPTION

A description is given below of a display apparatus and a method for changing an area by the display apparatus according to embodiments of the present disclosure, with reference to the attached drawings.

Outline of Recognition Group Rectangle and Neighborhood Rectangle

FIGS.1A and1Bare diagrams schematically illustrating a recognition group rectangle101and neighborhood rectangles102A and102B (collectively referred to as the neighborhood rectangle102when not distinguished from each other). The neighborhood rectangles102A and102B are examples of a determination area. In the present embodiment, the recognition group rectangle101is a circumscribed rectangle of one or more stroke data. The neighborhood rectangle102is an area for determining whether or not to include stroke data to be handwritten next in a recognition group of the previous stroke data. The recognition group is a group of stroke data forming hand drafted data to be collectively recognized as one or more characters. The neighborhood rectangle102is set based on the recognition group rectangle101, and the stroke data in the recognition group rectangle101and stroke data in the neighborhood rectangle102belong to the same recognition group.

One of the features of the display apparatus according to the present embodiment is setting the neighborhood rectangle102differently depending on whether a time T has elapsed from a pen-up (whether elapsed time from separation of an input device from a touch panel exceeds a threshold). The time T (threshold) is predetermined and may be stored in a memory, for example, by a manufacturer. Alternatively, the time T may be set by a user. “Pen-up” means that the input device having been in contact with a display (touch panel) is separated from the display (i.e., a pen lift event). “Pen-up” corresponds to disengaging the writing mode for inputting a stroke.

“The time T has elapsed from the pen-up” means that no hand drafted input by the user is detected during the time T.

“The time T has not elapsed from the pen-up” means that hand drafted input by the user is detected in the time T. “The time T has not elapsed from the pen-up” may be referred to as “in successive input” or “successively inputting.”

FIG.1Aillustrates an example of the neighborhood rectangle102A for a case where the time T has elapsed from the pen-up. InFIG.1A, the time T has elapsed after a character104“” (a Japanese hiragana character pronounced as “u”) is handwritten. The neighborhood rectangle102A in this case is set as follows.

Height: height H of the recognition group rectangle101

Width: height H of the recognition group rectangle101+α

In other words, when the time T has elapsed from the pen-up, on the assumption of horizontal writing in Japanese, the neighborhood rectangle102A has a height same as that of the character already handwritten (an upper end and a lower end same as those of the recognition group rectangle101) and a width by the size of next character. The width of the neighborhood rectangle102A extends in the rightward direction from the right end of the recognition group rectangle101. The value “α” is added as a margin (offset) to the neighborhood rectangle102A. In other words, the right end of the neighborhood rectangle102is shifted by the height H+the value α from the right end of the recognition group rectangle101. The offset has a distance (including a distance of 0) and is also referred to as a margin value, a correction value, an adjustment value, an additional value, or a surplus value. Note that, in the state illustrated inFIG.1A, the handwritten character104“” may be character-recognized or may remain as stroke data.

FIG.1Billustrates an example of the neighborhood rectangle102B (a surrounding area) for a case where the time T has not elapsed from the pen-up (in successive input). InFIG.1A, the time T has not yet elapsed from the pen-up after hand drafted input of the character104“” The neighborhood rectangle102B during such successive input is set as follows.

The neighborhood rectangle102B is set with respect to the current recognition group rectangle101as follows.

Upper: a value β1,

Lower: width W of recognition group rectangle101+β3,

Right: height H of recognition group rectangle101+β4

Each of the values β1to β4are margins (offset) to the neighborhood rectangle102B. In other words, the upper end, the lower end, the left end, and the right end of the neighborhood rectangle102are respectively shifted (offset) from those of the recognition group rectangle101by the value β1, the width W+the value β3, the value β2, and the height H+the value β4. In other words, the neighborhood rectangle102B is an area generated by enlarging the current recognition group rectangle101upward, downward, leftward, and rightward.

By differently setting the neighborhood rectangle102in this manner, a display apparatus2(seeFIGS.2A to2C) according to the present embodiment easily recognizes a character or the like intended by the user from the stroke data. When the user adds a character to the right of the character104“,” the user often expects the display apparatus to collectively recognize the character104“” and the added character. By contrast, when another character is handwritten, for example, below the character104“,” it can be assumed that the user has performed line feed. In this case, the user often handwrites a character string unrelated to the previous character104. In such a case, there is a high possibility that the character104and the character below the character104are erroneously character-recognized as a character string different from what intended by the user, if collectively recognized. In other words, if the user is not in successive input, it is preferable to limit the neighborhood rectangle102to the right of the recognition group rectangle101. In this specification, to the right of the recognition group rectangle101indicates a rightward direction of the recognition group rectangle101displayed on the screen as viewed from the user facing the screen. The same applies to the left, the upper, and the lower. The screen may be integral with the touch panel.

However, in successive input, the position at which the user handwrites the next stroke with respect to the stroke data being written varies depending on the stroke order of the character. For example, in the case of a Japanese hiragana character “” the third stroke is handwritten on the left of the second stroke, and the third stroke should also be character-recognized as a part of “” As another example, an upper dot (superscript dot) included in an alphabet of “i” or “j” is handwritten above the first stroke depending on the stroke order, but the character including the upper dot should be character-recognized. In this way, depending on the character, stroke data is also added above or to the left of the recognition group rectangle101(in many cases, stroke data is added below or to the right of the recognition group rectangle). In other words, in successive input, it is preferable to expand the neighborhood rectangle102around the recognition group rectangle101.

In this way, the stroke data of the same recognition group (a plurality of stroke data to be recognized collectively) differs depending on whether or not successive input is being performed. In the present embodiment, since the display apparatus differently sets the handwriting area (the neighborhood rectangle102) of the stroke data to be included in the same recognition group in accordance with the time T from the pen-up, the neighborhood rectangle102can be appropriately set.

Terms

“Input device” may be any means with which a user inputs handwriting (hand drafting) by designating coordinates on a touch panel. Examples thereof include a pen, a human finger, a human hand, and a bar-shaped member.

A series of user operations including engaging a writing mode, recording movement of an input device or portion of a user, and then disengaging the writing mode is referred to as a “stroke.” The engaging of the writing mode may include, if desired, pressing an input device against a display or screen, and disengaging the writing mode may include releasing the input device from the display or screen. Alternatively, a stroke includes tracking movement of the portion of the user without contacting a display or screen. In this case, the writing mode may be engaged or turned on by a gesture of a user, pressing a button by a hand or a foot of the user, or otherwise turning on the writing mode, for example using a pointing device such as a mouse. The disengaging of the writing mode can be accomplished by the same or different gesture used to engage the writing mode, releasing the button, or otherwise turning off the writing mode, for example using the pointing device or mouse. “Stroke data” is data displayed on a display based on a trajectory of coordinates of a stroke input with the input device. Such stroke data may be interpolated appropriately. “Hand drafted data” is data having one or more stroke data. “Hand drafted data” is data used for displaying (reproducing) a screen image including objects handwritten or hand-drafted by the user. “Hand drafted input” relates to a user input such as handwriting, drawing, and other forms of input. The hand drafted input may be performed via touch interface, with a tactile object such as a pen or stylus or with the user's body. The hand drafted input may also be performed via other types of input, such as gesture-based input, hand motion tracking input or other touch-free input by a user.

A character string obtained by character recognition and conversion from hand drafted data may include, in addition to text data, data displayed based on a user operation, such as a stamp of a given character or mark such as “complete,” a graphic such as a circle or a star, or a line. Characters include numbers, alphabets, symbols, and the like. A character is also referred to as text data.

“Collectively recognizing a plurality of stroke data” means that the entirety of the plurality of stroke data is recognized as a character string appropriate as a whole. The character string after recognition may include a single character or a plurality of characters.

Configuration of Apparatus

Referring toFIGS.2A to2C, a description is given of a general arrangement of the display apparatus2according to the present embodiment.FIGS.2A to2Care diagrams illustrating examples of general arrangement of the display apparatus2.FIG.2Aillustrates, as an example of the display apparatus2, an electronic whiteboard having a landscape-oriented rectangular shape and being hung on a wall.

As illustrated inFIG.2A, the display apparatus2includes a display220(a screen). A user U handwrites (also referred to as “inputs” or “draws”), for example, a character on the display220using a pen2500.

FIG.2Billustrates, as another example of the display apparatus2, an electronic whiteboard having a portrait-oriented rectangular shape and being hung on a wall.

FIG.2Cillustrates, as another example, the display apparatus2placed on the top of a desk230. It is not necessary to adjust the height of the desk230, which is a general-purpose desk, when the display apparatus2having a thickness of about 1 centimeter is placed thereon. Further, the display apparatus2is portable and easily moved by the user.

Examples of an input method of coordinates by the pen2500include an electromagnetic induction method and an active electrostatic coupling method. In other example, the pen2500further has functions such as drawing pressure detection, inclination detection, a hover function (displaying a cursor before the pen is brought into contact), or the like.

Hardware Configuration

A description is given of a hardware configuration of the display apparatus2according to the present embodiment, with reference toFIG.3. The display apparatus2has a configuration of an information processing apparatus or a computer as illustrated in the drawing.FIG.3is a block diagram illustrating an example of the hardware configuration of the display apparatus2. As illustrated inFIG.3, the display apparatus2includes a central processing unit (CPU)201, a read only memory (ROM)202, a random access memory (RAM)203, and a solid state drive (SSD)204.

The CPU201controls entire operation of the display apparatus2. The ROM202stores a control program such as an initial program loader (IPL) to boot the CPU201. The RAM203is used as a work area for the CPU201.

The SSD204stores various data such as an operating system (OS) and a control program for display apparatuses. This program may be an application program that runs on an information processing apparatus equipped with a general-purpose operating system (OS) such as WINDOWS, MAC OS, ANDROID, and IOS. In this case, the display apparatus2is usually used as a general-purpose information processing device. However, when a user executes an installed application program, the display apparatus2receives handwriting or the like performed by the user similarly to a dedicated display apparatus.

The display apparatus2further includes a display controller213, a touch sensor controller215, a touch sensor216, a tilt sensor217, a serial interface218, a speaker219, the display220, a microphone221, a wireless communication device222, an infrared interface (I/F)223, a power control circuit224, an alternating current (AC) adapter225, a battery226, and a power switch227.

The display controller213controls display of an image for output to the display220, etc. The touch sensor216detects that the pen2500, a user's hand or the like is brought into contact with the display220. The pen or the user's hand is an example of input device. The touch sensor216also receives a pen identifier (ID).

The touch sensor controller215controls processing of the touch sensor216. The touch sensor216performs coordinate input and coordinate detection. More specifically, in a case of optical sensing, for inputting and detecting coordinates, the display220is provided with two light receiving and emitting devices disposed on both upper side ends of the display220, and a reflector frame surrounding the sides of the display220. The light receiving and emitting devices emit a plurality of infrared rays in parallel to a surface of the display220. Light-receiving elements receive lights passing in the direction that is the same as an optical path of the emitted infrared rays, which are reflected by the reflector frame. The touch sensor216outputs position information of the infrared ray that is blocked by an object after being emitted from the two light receiving and emitting devices, to the touch sensor controller215. Based on the position information of the infrared ray, the touch sensor controller215detects a specific coordinate that is touched by the object. The touch sensor controller215further includes a communication circuit215afor wireless communication with the pen2500. For example, when communication is performed in compliance with a standard such as BLUETOOTH (registered trademark), a commercially available pen can be used.

When one or more pens2500are registered in the communication circuit215ain advance, the display apparatus2communicates with the pen2500without connection setting between the pen2500and the display apparatus2, performed by the user.

The power switch227turns on or off the power of the display apparatus2. The tilt sensor217detects the tilt angle of the display apparatus2. The tilt sensor217is mainly used to detect whether the display apparatus2is being used in any of the states inFIG.2A,2B, or2C. For example, the display apparatus2automatically changes the thickness of characters or the like depending on the detected state.

The serial interface218is a communication interface to connect the display apparatus2to extraneous sources such as a universal serial bus (USB). The serial interface218is used to input information from extraneous sources. The speaker219is used to output sound, and the microphone221is used to input sound. The wireless communication device222communicates with a terminal carried by the user and relays the connection to the Internet, for example.

The wireless communication device222performs communication in compliance with Wi-Fi, BLUETOOTH (registered trademark) or the like. Any suitable standard can be applied other than the Wi-Fi and BLUETOOTH (registered trademark). The wireless communication device222forms an access point. When a user sets a service set identifier (SSID) and a password that the user obtains in advance in the terminal carried by the user, the terminal is connected to the access point.

It is preferable that two access points are provided for the wireless communication device222as follows:

(a) Access point to the Internet; and (b) Access point to Intra-company network to the Internet. The access point (a) is for users other than, for example, company staffs. The access point (a) does not allow access from such users to the intra-company network but allow access to the Internet. The access point (b) is for intra-company users and allows such users to access the intra-company network and the Internet.

The infrared I/F223detects an adjacent display apparatus2. The infrared I/F223detects an adjacent display apparatus2using the straightness of infrared rays. Preferably, one infrared I/F223is provided on each side of the display apparatus2. This configuration allows the display apparatus2to detect the direction in which the adjacent display apparatus2is disposed. Such arrangement extends the screen. Accordingly, the user can instruct the adjacent display apparatus2to display a previous handwritten object. That is, one display220(screen) corresponds to one page, and the adjacent display220displays the handwritten object on a separate page.

The power control circuit224controls the AC adapter225and the battery226, which are power supplies for the display apparatus2. The AC adapter225converts alternating current shared by a commercial power supply into direct current.

In a case where the display220is a so-called electronic paper, the display220consumes little or no power to maintain image display. In such case, the display apparatus2may be driven by the battery226. With this structure, the display apparatus2is usable as, for example, a digital signage in places such as outdoors where power supply connection is not easy.

The display apparatus2further includes a bus line210. The bus line210is an address bus or a data bus that electrically connects the elements illustrated inFIG.3, such as the CPU201, to each other.

The touch sensor216is not limited to the optical type. In another example, the touch sensor216is a different type of detector, such as a capacitive touch panel that identifies the contact position by detecting a change in capacitance, a resistance film touch panel that identifies the contact position by detecting a change in voltage of two opposed resistance films, or an electromagnetic induction touch panel that identifies the contact position by detecting electromagnetic induction caused by contact of an object to a display. The touch sensor216can be of a type that does not require an electronic pen to detect whether the pen tip is in contact with the surface of the display220. In this case, a fingertip or a pen-shaped stick is used for touch operation. In addition, the pen2500can have any suitable shape other than a slim pen shape.

Functions

A description is now given of a functional configuration of the display apparatus2according to the present embodiment, with reference toFIG.4.FIG.4is a block diagram illustrating an example of the functional configuration of the display apparatus2according to the present embodiment. The display apparatus2includes an input receiving unit21, a drawing data generation unit22, a character recognition unit23, a display control unit24, a data recording unit25, a network communication unit26, an operation receiving unit27, an area setting unit28, and an exclusion unit29. The functional units of the display apparatus2are implemented by or are caused to function by operation of any of the elements illustrated inFIG.3according to an instruction from the CPU201according to a program loaded from the SSD204to the RAM203.

The input receiving unit21receives input of stroke data (coordinate point sequence) by detecting coordinates of a position at which an input device such as the pen2500contacts the touch sensor216. The drawing data generation unit22acquires the coordinates (i.e., contact coordinates) of the position touched by the pen tip of the pen2500from the input receiving unit21.

The drawing data generation unit22connects a plurality of contact coordinates into a coordinate point sequence by interpolation, to generate stroke data.

The character recognition unit23performs character recognition processing on one or more stroke data (hand drafted data) input by the user and converts the stroke data into one or more character codes. The character recognition unit23recognizes characters (of multilingual languages such as English as well as Japanese), numbers, symbols (e.g., %, $, and &), graphics (e.g., lines, circles, and triangles) concurrently with a pen operation by the user. Although various algorithms have been proposed for the recognition method, a detailed description is omitted on the assumption that known techniques are used in the present embodiment.

The display control unit24displays, on the display220, for example, hand drafted data, a character string converted from the hand drafted data, and an operation menu to be operated by the user. The data recording unit25stores hand drafted data input on the display apparatus2, a converted character string, a screenshot on a personal computer (PC) screen, a file, and the like in a storage area40. The network communication unit26connects the wireless communication device222to a network such as a local area network (LAN) and transmits and receives data to and from other devices via the network.

The area setting unit28sets the neighborhood rectangle102for determining whether stroke data is to be included in the recognition group differently depending on whether the time T has elapsed after the input device is separated from the touch panel.

When the stroke data received by the input receiving unit21satisfies a predetermined condition, the exclusion unit29excludes, from the recognition group, even stroke data contained in the neighborhood rectangle102.

The display apparatus2includes the storage area40implemented by, for example, the SSD204or the RAM203illustrated inFIG.3. The storage area40includes a stroke data storage area41.

FIGS.5A,5B, and5Care diagrams illustrating data related to stroke data stored in the stroke data storage area41.FIG.5Ais a conceptual diagram illustrating page data. The page data is data of one page displayed on the display220. As illustrated inFIG.5A, the page data includes data items of page data ID for identifying a page, start time indicating the time at which display of the page is started, end time indicating the time at which hand drafted input to the page is stopped, and stroke arrangement data ID for identifying data on an arrangement of strokes made by an input device, in association with one another. InFIG.5A, text data, image data, tables, graphics, and the like after character recognition are omitted.

The stroke arrangement data is to be used for displaying one stroke data on the display220. For example, when the user draws an alphabet “S” with an input device in one stroke, one stroke data ID is assigned to the alphabet “S” to be identified. When the user draws an alphabet “T” with an input device in two strokes, two stroke data IDs are assigned to the alphabet “T” to be identified.

The stroke arrangement data includes detailed information as illustrated inFIG.5B.FIG.5Bis a conceptual diagram illustrating a data structure of the stroke arrangement data. One stroke arrangement data includes a plurality of stroke data. Each stroke data includes a stroke data ID for identifying that stroke data, start time when drawing of that stroke starts, end time when drawing of that stroke ends, a color of the stroke, a width of the stroke, a recognition group, type, and a coordinate array data ID for identifying arrangement of points of the stroke.

The recognition group is a group of stroke data forming hand drafted data to be collectively character-recognized as one or more characters. The type indicates the determined type, for example, a character or a graphic, to which the stroke data belongs. Types also include English cursive.

Further, the coordinate array data includes detailed information as illustrated inFIG.5C.FIG.5Cis a conceptual diagram illustrating a data structure of the coordinate array data. As illustrated inFIG.5C, the coordinate array data includes coordinates (X coordinate value and Y coordinate value) of a point on the display220, time difference (in milliseconds) from the start of drawing of the stroke to when the stroke passes that point, and drawing pressure by the pen2500on that point. That is, one coordinate array data inFIG.5Bis a collection of one-point data inFIG.5C. For example, in case in which a user draws the alphabet “S” with the input device in one stroke, the stroke passes a plurality of points. Accordingly, the coordinate array data indicates those passing points in drawing of the stroke.

Condition Under Which Stroke Data Belongs to Same Recognition Group

Referring toFIGS.6and7, the conditions for the same recognition group will be described in detail. The area setting unit28sets the determination area (for determining whether stroke data is to be included in the recognition group) differently depending on whether input of stroke data is continuously received. In other words, the determination area for determining whether to include the stroke data in the recognition group is differently set depending on whether or not the predetermined time has elapsed after the input device is separated from the touch panel.

A description is given of a case where the time T has not elapsed from a pen-up (in successive input).

FIG.6is a diagram illustrating the same recognition group in a case where the time T has not elapsed from the pen-up. The circumscribed rectangle of one or more strokes in successive input is the recognition group rectangle101. The area of the neighborhood rectangle102B in successive input is defined with respect to the recognition group rectangle101as follows.

The upper end of the neighborhood rectangle102B is shifted upward from the upper end of the recognition group rectangle101by the value β1. The left end of the neighborhood rectangle102B is shifted leftward from the left end of the recognition group rectangle101by the value β2. The lower end of the neighborhood rectangle102B is shifted downward from the lower end of the recognition group rectangle101by the width W of the recognition group rectangle101plus the value133. The right end of the neighborhood rectangle102B is shifted rightward from the lower end of the recognition group rectangle101by the height H of the recognition group rectangle101plus the value β4.

Stroke data having a portion protruding from the neighborhood rectangle102is determined as having been handwritten in the neighborhood rectangle102when the proportion of the protruding portion is equal to or less than a threshold. Stroke data handwritten in the recognition group rectangle101may or may not be regarded as being contained in the neighborhood rectangle102.

Therefore, the stroke data in the recognition group rectangle101and the stroke data in the neighborhood rectangle102belong to the same recognition group.

The margins are, for example, β1-β2-1.5 cm, and β3-β4-2 cm. When the width W of the recognition group rectangle101is 1.5 cm and the height H thereof is 0.5 cm, the width and height of the neighborhood rectangle102B are as follows.

Width: the value β2+the width W of the recognition group rectangle101+the height H of recognition group rectangle101+the value β4=5.5 cm

Height: the value β1+the height H of the recognition group rectangle101+the width W of the recognition group rectangle101+the value β3=5.5 cm

The margins vary depending on the size of the display220, the number of pixels, and the intended use. The above-described margins are examples in a case where hand drafted data has a size sharable by several persons on the display220of about 40 inches and 2880×2160 pixels. The same applies to a case where stroke is input in a manner different from successive input.

The values β1and β2are respectively added upward and leftward to the recognition group rectangle101as margins for receiving handwriting of stroke data in order to recognize the following stroke data. Japanese characters are often written in the downward direction or rightward direction. However, there are Japanese characters (e.g., “” pronounced as “hu”) in which a stroke is drawn on the left of the previous stroke, and there are characters (e.g., “i” and “j”) in which a stroke is drawn above the previous stroke. Therefore, the neighborhood rectangle102is enlarged leftward and upward directions by the value β1and the value β2, respectively.

The margin for receiving handwriting of stroke data is provided on the right of the recognition group rectangle101considering the characteristics of construction of Chinese characters. Specifically, for example, in a case where the user successively draws a stroke on the right of “” (a left part of a Chinese character), the height of “” is assumed to be the character size, and the neighborhood rectangle102B is enlarged by the size of one character in the rightward direction.

The margin is provided below the recognition group rectangle101considering characteristics of construction of Chinese characters. For example, in a case where the user successively draws a stroke below “” (an upper part of a Chinese character), the width of “” is assumed to be the character size, and the neighborhood rectangle102B is enlarged by the size of one character in the downward direction.

A description is given of a case where the time T has elapsed from a pen-up.

FIG.7is a diagram illustrating the same recognition group in a case where the time T has elapsed from the pen-up. The circumscribed rectangle of one or more stroke data sequentially input within the time T from a pen-up is the recognition group rectangle101. The area of the neighborhood rectangle102A for the case where the time T has elapsed is defined with respect to the recognition group rectangle101as follows.

Height: height H of the recognition group rectangle101

Width: height H of the recognition group rectangle101+α from the right end of the recognition group rectangle101

When the time T has elapsed from the pen-up, on the assumption of the character size of Japanese horizontal writing, the display apparatus2sets the neighborhood rectangle102A extending in the rightward direction by one character size. Specifically, the area setting unit28expands the neighborhood rectangle102A in the rightward direction by the value α on the assumption that the user handwrites a stroke rightward with a blank space from the recognition group rectangle101. The area setting unit28determines only the rightward area of the circumscribed rectangle (the recognition group rectangle101) of the one or more already-input stroke data as the determination area (the neighborhood rectangle102A) for determining whether to include the next stroke data in the recognition group.

The display apparatus2determines that a Japanese character106“” (pronounced as “o”) in the recognition group rectangle101and the stroke data in the neighborhood rectangle102A belong to the same recognition group.

The value α is, for example, 3 cm. When the recognition group rectangle101has a width of 4 cm and a height of 6 cm, the neighborhood rectangle102A has the following width and height.

Width: height H of the recognition group rectangle101+the value α=9 cm

Height: Height H of the recognition group rectangle101=6 cm

As described above, the area setting unit28differently sets the determination area (the neighborhood rectangle102) for determining whether to include next stroke data in the recognition group depending on whether or not the time T has elapsed after the input device is separated from the touch panel.

A description is given of conditions under which stroke data is not included in the same recognition group.

There are not a few cases where it is not desired to include stroke data handwritten in the neighborhood rectangle102in the same recognition group as that of previous stroke data. A detailed description of this is given below, with reference toFIG.8.FIG.8illustrates a character110and a table120handwritten on a screen. The user may handwrite a frame line of the table120while handwriting the character110, that is, before the time T elapses from the handwriting of the character110. In this case, it is difficult for the display apparatus2to separate the character110from the table120and perform character recognition on the character110. That is, the frame line of the table120may be erroneously character-recognized together with the character110“” (Japanese hiragana character pronounced as “a”).

In some cases, the user writes an explanatory text111outside the table120and draws with strokes an arrow113pointing at a graphic112or the like in the table120. In this case, the arrow113is included in the neighborhood rectangle102(seeFIG.7) after the time T has elapsed from the handwriting of the explanatory text111“” (Japanese meaning “this”). Accordingly, there is a possibility that the explanatory text111“” and the arrow113are collectively character-recognised.

Therefore, in the case ofFIG.8, it is preferable not to include the stroke data in the neighborhood rectangle102in the same recognition group.

Therefore, the exclusion unit29of the present embodiment determines a condition under which the stroke data is not included in the same recognition group as that of previous stroke data as follows.

FIG.9is a diagram illustrating a condition under which stroke data is not included in the same recognition group. The exclusion unit29excludes, from the same recognition group, stroke data that is contained in the neighborhood rectangle102but satisfies excluding condition (i) or (ii) presented below.

(i) the stroke data has a height larger than a threshold value a; and

(ii) the stroke data has a width larger than a threshold value b and a height smaller than a threshold value c smaller than the threshold value a.

The threshold value a (an example of a first threshold value) and the threshold value b (an example of a second threshold value) are, for example, 9 cm. The threshold value c (an example of a third threshold value) is, for example, 2.5 cm. These threshold values vary depending on the size of the display220, the number of pixels of the display220, how many people share characters, and the like.

The excluding condition (i) is for setting the threshold value a as the maximum height of a character and determining that stroke data exceeding the threshold value a is a graphic. The excluding condition (ii) is for determining that stroke data having a width exceeding the threshold value b is a graphic. The threshold value b is the maximum width of a general character. Further, the excluding condition (ii) is for including English cursive in one recognition group.

A description is given of determining whether stroke data belongs to the same recognition group using regions R1to R4divided by threshold values a, b, and c inFIG.9.

Stroke data entirely contained in the regions R1and R2does not satisfy the excluding conditions (i) and (ii) and is assumed to be a Japanese character. Accordingly, the stroke data entirely contained in the regions R1and R2is not excluded from the same recognition group.

Stroke data entirely contained in the regions R1, R2, and R3does not satisfy the excluding conditions (i) and (ii) and is assumed to be a Japanese character. Accordingly, the stroke data entirely contained in the regions R1, R2, and R3is not excluded from the same recognition group. These conditions cope with English cursive. Specifically, stroke data of cursive characters such as “English” handwritten in one stroke is not excluded from the same recognition group (is not regarded as a graphic), and thus the display apparatus2recognizes the stroke data as characters. The display apparatus2may recognize stroke data entirely contained in the regions R1, R2, and R3as English cursive.

Stroke data entirely contained in the regions R2and R4satisfies the excluding condition (ii) and is assumed to be a graphic (for example, a horizontal line). Accordingly, the stroke data entirely contained in the regions R2and R4is excluded from the same recognition group.

Stroke data entirely contained in the regions R1to R4does not satisfy the excluding conditions (i) and (ii) and is assumed to be a Japanese character. Accordingly, the stroke data entirely contained in the regions R1to R4is not excluded from the same recognition group. Also in this case, English cursive can be recognized.

As described above, depending on whether or not the stroke data satisfies the excluding condition (i) or (ii), the exclusion unit29forcibly determines the stroke data contained in the neighborhood rectangle102as not belonging to the same recognition group. Thus, even when a graphic and a character are handwritten in a mixed manner, the character recognition unit23recognizes the character by separating the character from the graphic.

In addition to the excluding conditions (i) and (ii), there are following excluding conditions for the same recognition group as presented below.

Excluding condition 1: The stroke data is not contained in the neighborhood rectangle102.

Excluding condition 2: An immediately preceding operation with the pen2500in use includes processing, such as “character conversion,” other than stroke drawing.

Excluding condition 3: In a special example such as area control, stroke data is determined as being input in another area.

Excluding condition 4: The pen type is different.

A description is given below of a sequence of operations.

FIG.10is a flowchart illustrating a procedure in which the character recognition unit23determines stroke data of the same recognition group. The process ofFIG.10is repeatedly executed while the display apparatus2is on.

The display apparatus2receives input of a stroke (proceeding stroke) relative to which whether a subsequent stroke is grouped in the same recognition group is determined (S1). The input receiving unit21detects coordinates of the points touched by the input device, and the drawing data generation unit22generates stroke data. The display control unit24controls the display220to display the stroke data. The exclusion unit29determines whether or not the stroke data satisfies the excluding condition (i) or (ii) under which the stroke data does not belong to the same recognition group (seeFIG.9). Stroke data that disagrees with the excluding condition (i) or (ii) under which the stroke data does not belong to the same recognition group is subjected to subsequent processing. The determination of step S1will be described with reference to the flowchart ofFIG.11.

The area setting unit28determines whether or not the time T has elapsed from a pen-up after completion of input of the stroke from which the stroke data is generated in step S1(S2).

In a state where the time T has not elapsed (Yes in S2), the display apparatus2receives input of a stroke (continuous input) (S3). The input receiving unit21detects the coordinates of the points touched by the input device, and the drawing data generation unit22generates stroke data. The display control unit24controls the display220to display the stroke data. The exclusion unit29determines whether or not the stroke data of S3satisfies the above-described excluding condition (i) or (ii) under which the stroke data does not belong to the same recognition group. Stroke data that disagrees with the excluding condition (i) or (ii) is subjected to subsequent processing.

The area setting unit28sets the neighborhood rectangle102B illustrated inFIG.6for successive input, based on the stroke data of step S1, and determines whether the stroke data of step S3is contained in the neighborhood rectangle102(S4).

When the stroke data of step S3is determined as being contained in the neighborhood rectangle102B (Yes in S4), the area setting unit28determines that the stroke data of step S1and the stroke data of S3belong to the same recognition group (S5).

When the stroke data of step S3is not contained in the neighborhood rectangle102B (No in S4), the area setting unit28determines that the stroke data of step S1and the stroke data of S3do not belong to the same recognition group, that is, exclude the stroke data of S3from the recognition group of stroke data of step S1(S6).

In a state where the elapsed time from the pen-up after the handwriting of the stroke in step S1exceeds the time T (No in S2), the display apparatus2receives input of a stroke (S7). The input receiving unit21detects coordinates of the points touched by the input device, and the drawing data generation unit22generates stroke data. The display control unit24controls the display220to display the stroke data. The exclusion unit29determines whether or not the stroke data satisfies the excluding condition (i) or (ii) under which the stroke data does not belong to the same recognition group. Stroke data that disagrees with the excluding condition (i) or (ii) under which the stroke data does not belong to the same recognition group is subjected to subsequent processing.

Next, the area setting unit28sets the neighborhood rectangle102A illustrated inFIG.7for the case where the time T has elapsed, based on the stroke data of step S1, and determines whether or not the stroke data of step S7is contained in the neighborhood rectangle102A (S8).

When the stroke data of step S7is determined as being contained in the neighborhood rectangle102A (Yes in S8), the area setting unit28determines that the stroke data of step S1and the stroke data of S7belong to the same recognition group (S5).

When the stroke data of step S7is not contained in the neighborhood rectangle102A (No in S8), the area setting unit28determines that the stroke data of step S1and the stroke data of S7do not belong to the same recognition group, that is, exclude the stroke data of S7from the recognition group of the stroke data of step S1(S6).

FIG.11is a flowchart for determining whether the stroke data satisfies the excluding condition (i) or (ii) under which the stroke data does not belong to the same recognition group, described in steps S1, S3, and S7inFIG.10.

The display apparatus2receives input of a stroke with an input device (e.g., an electronic pen) (S11). The input receiving unit21detects coordinates of the points touched by the input device, and the drawing data generation unit22generates stroke data. The display control unit24controls the display220to display the stroke data.

The exclusion unit29determines whether or not the height of the stroke data is larger than the threshold value a (S12).

When the height of the stroke data is equal to or smaller than the threshold value a (No in S12), the exclusion unit29determines whether the width of the stroke data in step S11is larger than the threshold value b and the height thereof is smaller than the threshold value c (S13).

In a case where the determination of either step S12or step S13is Yes, the exclusion unit29excludes the stroke data of step S11from the same recognition group (S14).

When the determination in step S13is No, the area setting unit28determines that the stroke data of step S11is to be subjected to the determination of the same recognition group. That is, the area setting unit28determines whether or not the stroke data of step S11is contained in the neighborhood rectangle102in the process ofFIG.10.

Example of Character Recognition Result

Referring toFIG.12, a description is given of a recognition result of hand drafted data.FIG.12illustrates an example of an operation guide500provided by the display apparatus2and selectable candidates530displayed by the operation guide500. The operation guide500is displayed after elapse of a predetermined time from the pen-up after the user inputs a handwritten object504. Specifically, as the user handwrites a stroke, the input receiving unit21detects coordinates of the stroke, and the drawing data generation unit22generates stroke data based on the trajectory of coordinates. When the user handwrites a plurality of strokes, hand drafted data having one or more stroke data is generated. The predetermined time may be different from or the same as the time T.

The operation guide500displays an operation command candidate510“” (pronounced as “ohaio-shiten ni soushin” and meaning “send to Ohio branch”), a recognized character string candidate506“” (Japanese hiragana character string pronounced as “oha”), converted character string candidates507, and predicted converted-character string candidates508. The selectable candidates530includes the recognized character string candidate506, the converted character string candidates507, the predicted converted-character string candidates508, and the operation command candidate510. The selectable candidates530other than the operation command candidate510are referred to as character string candidates539. The recognized character string candidate506is an example of the result of the character recognition.

The handwritten object504is characters “” (Japanese hiragana characters, pronounced as “oha”) handwritten by the user. That is, with the neighborhood rectangle102of the present embodiment, the characters “” is determined as belonging to the same recognition group. The display apparatus2displays a rectangular handwriting area enclosure503enclosing the handwritten object504of the same recognition group. In the example illustrated inFIG.12, the operation guide500is displayed in response to input of two characters as an example, but the time of display thereof is not limited thereto. The operation guide500is displayed in response to suspension of handwriting by the user. One or more stroke data included in the same recognition group until the interruption are collectively character-recognized to one or more characters. Thus, the number of characters in the handwritten object504is any number.

As each of the recognized character string candidate506, the converted character string candidates507, and the predicted converted-character string candidates508, one or more candidates are arranged in descending order of probability. The recognized character string candidate506“” (Japanese hiragana characters, pronounced as “oha”) is a candidate of recognition result. In this example, the character recognition unit23has correctly recognized “.”

The converted character string candidates507are results of kana-kanji conversion (e.g., Japanese katakana characters “” pronounced as “oha,” a mixture of kanji and hiragana characters “” pronounced as “owa” and meaning “tail is,” or kanji characters “” pronounced as “owa”) of the recognized character string candidate506. Alternatively, the converted character string candidates507are results of converted character string candidates (for example, idioms including “”) converted from the result of kana-kanji conversion. The predicted converted-character string candidates508are candidates predicted from the converted character string candidates507, respectively. In this example, “” (Japanese character string pronounced as “ohayou no aisatsu” and meaning “morning greeting”) and “” (Japanese character string pronounced as “o wa kuroi” and meaning “tail is black”) are displayed.

The operation command candidate510is a predefined operation command candidate (command such as file operation or text editing) displayed in accordance with the recognized character. In the example ofFIG.12, a line head character “>>”511is an indication of the operation command candidate. In the example inFIG.12, the recognized character string candidate506“” (pronounced as “oha”) partially matches definition data, and the corresponding operation command candidate510is displayed.

The operation command candidate510is displayed when the operation command definition data including the converted character string is found and is not displayed in the case of no-match.

The operation guide500includes an operation header520including buttons501,502,505, and509. The button501is a graphical representation for receiving an operation of switching between predictive conversion and kana conversion. The button502is a graphical representation for receiving page operation of the candidate display. In the example illustrated inFIG.12, there are three candidate display pages, and the first page is currently displayed. The button505is a graphical representation for receiving closing of the operation guide500. When the operation receiving unit27receives pressing by the user of the button505, the display control unit24deletes the displayed objects other than the handwritten object. The button509is a graphical representation for receiving batch deletion of the display. When the operation receiving unit27receives pressing by the user of the button509, the display control unit24deletes the displayed objects illustrated inFIG.12, thereby enabling the user to perform hand drafted input from the beginning.

The display apparatus2of the present embodiment focuses on the fact that a plurality of stroke data to be collectively recognized differs depending on whether or not successive input is being performed. The display apparatus2sets the neighborhood rectangle102(for determining stroke data to be included in the same recognition group) differently depending on whether the time T has elapsed. Thus, the neighborhood rectangle102is appropriately set.

In addition, the display apparatus2of the present embodiment excludes, from the same recognition group, stroke data that is contained in the neighborhood rectangle102but matches the excluding condition under which the stroke data does not belong to the same recognition group. Accordingly, the display apparatus2accurately recognizes characters even when the user handwrites characters and graphics in a mixed manner.

In this embodiment, a description is given of a display system19including a server12as an example of an information processing apparatus that performs character recognition.

FIG.13is a schematic diagram illustrating an example of a configuration of the display system19according to the present embodiment. The display apparatus2and the server12are connected to each other through a network such as the Internet.

In the display system19, the display apparatus2includes the input receiving unit21, the drawing data generation unit22, the display control unit24, the network communication unit26, and the operation receiving unit27illustrated inFIG.4.

By contrast, the server12includes the character recognition unit23, the data recording unit25, the area setting unit28, the exclusion unit29, and a network communication unit.

The network communication unit26of the display apparatus2transmits the stroke data to the server12. The server12performs the same processing as in the flowcharts ofFIGS.10and11and transmits the recognition result to the display apparatus2.

As described above, in the display system19, the display apparatus2and the server12interactively process and display text data.

Now, descriptions are given of other application of the embodiments described above.

The present disclosure is not limited to the details of the embodiments described above, and various modifications and improvements are possible.

For example, in the above-described embodiments, the neighborhood rectangle102is set on the assumption of Japanese handwriting from left to right. For a language in which writing direction is not from left to right, the neighborhood rectangle102is set in accordance with the writing direction.

In the above-described embodiments, the stroke data is converted mainly into Japanese, but the conversion target language of the stroke data may be other languages (English, Chinese, Hindi, Spanish, French, Arabic, Russian, etc.).

In the description above, the display apparatus2being an electronic whiteboard is described as an example but is not limited thereto. A device having a substantially the same functions as the electronic whiteboard may be referred to as an electronic information board, an interactive board, or the like. The present disclosure is applicable to any information processing apparatus having a touch panel. Examples of the information processing apparatus with a touch panel include, but not limited to, a projector, an output device such as a digital signage, a head up display, an industrial machine, an imaging device, a sound collecting device, a medical device, a network home appliance, a laptop computer (personal computer or PC), a mobile phone, a smartphone, a tablet terminal, a game console, a personal digital assistant (PDA), a digital camera, a wearable PC, and a desktop PC.

Further, in the embodiments described above, the display apparatus2detects the coordinates of the tip of the pen on the touch panel by optical sensing. Alternatively, the display apparatus2may detect the coordinates of the pen tip by another method such as the above-mentioned method using ultrasonic waves. For example, the pen emits an ultrasonic wave in addition to the light, and the display apparatus2calculates a distance based on an arrival time of the sound wave. The display apparatus2determines the position of the pen based on the direction and the distance, and a projector draws (projects) the trajectory of the pen based on stroke data.

In the block diagram such asFIG.4, functional units are divided into blocks in accordance with main functions of the display apparatus2, in order to facilitate understanding the operation by the display apparatus2. Each processing unit or each specific name of the processing unit is not to limit a scope of the present disclosure. The processing implemented by the display apparatus2may be divided into a larger number of processing units depending on the content of the processing. In addition, a single processing unit can be further divided into a plurality of processing units.

One aspect of the present disclosure provides an information processing apparatus that includes circuitry to receive a plurality of stroke data respectively generated based on a plurality of strokes input by hand drafting, the plurality of stroke data including first stroke data and second stroke data being input after the first stroke data; set a determination area differently depending on whether an elapsed time from separation of the input device from the touch panel after input of the first stroke data exceeds a threshold, the determination area being for determining whether to include the second stroke data in a recognition group including the first stroke data: perform character recognition on the recognition group; and output a result of the character recognition.

Another aspect of the present disclosure provides a display system that includes a display apparatus and a server that communicates with the display apparatus. The display apparatus includes first circuitry to receive a plurality of stroke data input to a touch panel by an input device, the plurality of stroke data including first stroke data and second stroke data being input after the first stroke data; display the first stroke data and the second stroke data; transmit the plurality of stroke data to the server; and display, on a screen, a result of character recognition received from the server. The server includes second circuitry to set a determination area differently depending on whether an elapsed time from separation of the input device from the touch panel after input of the first stroke data exceeds a threshold, the determination area being for determining whether to include the second stroke data in a recognition group including the first stroke data; perform character recognition on the recognition group; and output the result of the character recognition to the display apparatus.