Mobile terminal and method of controlling the same

Provided is a mobile terminal including: a camera; a touch screen; and a controller that outputs to the touch screen a keypad that includes at least one graphic object of a first graphic object in conjunction with an optical character reader (OCR) function relating to the camera and a second graphic object in conjunction with a photographing function relating to the camera, in which the first and second graphic objects are together displayed on the keypad or one graphic object of the first and second graphic objects is displayed on the keypad, and in which, in a case where one graphic object of the first and second graphic objects is displayed on the keypad, based on application of a predetermined-type touch to the one graphic object, the controller changes one graphic object to the other graphic object.

CROSS-REFERENCE TO RELATED APPLICATION

Pursuant to 35 U.S.C. § 119(a), this application claims the benefit of earlier filing date and right of priority to Korean Application No. 10-2015-0117552, filed on Aug. 20, 2015, the contents of which is incorporated by reference herein in its entirety.

BACKGROUND OF THE DISCLOSURE

1. Field of the Disclosure

The present disclosure relates to a mobile terminal that increases its usage with user convenience in mind and a method of controlling the mobile terminal.

2. Background of the Disclosure

Efforts are ongoing to support and increase the functionality of mobile terminals. Such efforts include software and hardware improvements, as well as changes and improvements in the structural components.

In recent years, application of an optical character reader/recognition (OCR) function to a mobile terminal that is equipped with a camera has been actively studied. The OCR function is a function of reading characters (text) using light. The OCR function that is applied to the mobile terminal is a function in which light is emitted to letters, symbols, and numbers that are printed or hand-written on paper, the reflected right is converted into electrical signals and the electrical signals are input into the mobile terminal.

The OCR function may mean a function of performing image-to-text conversion. That is, the OCR function includes a function in which screen information (an image) that corresponds to text which is included in an image that is received through a camera is converted into text and the text that results from the conversion is input into the mobile terminal.

In addition, in recent years, with advanced camera technologies, a photographing function of capturing the image that is received through the camera has been widely used. Specifically, according to a user request, the mobile terminal captures the image that is received through the camera and inputs the captured image into the mobile terminal.

Accordingly, there is an increasing need for user interface (UI)/user experience (UX) through which information that a user wants is input in the form of text in an optimized way using the OCR function relating to the camera and the photographing function relating to the camera.

SUMMARY OF THE DISCLOSURE

Therefore, an aspect of the detailed description is to provide a mobile terminal that is capable of inputting information in an optimized way and a mobile terminal.

Another object of the detailed description is to provide a mobile terminal that provides a user interface through which information that a user wants can be input into an input field in an optimized way and a method of controlling the mobile terminal.

According to as aspect of the present invention, there is provided a mobile terminal including: a camera; a touch screen; and a controller that outputs to the touch screen a keypad that includes at least one graphic object of a first graphic object in conjunction with an optical character reader (OCR) function relating to the camera and a second graphic object in conjunction with a photographing function relating to the camera, in which the first and second graphic objects are together displayed on the keypad or one graphic object of the first and second graphic objects is displayed on the keypad, and in which, in a case where one graphic object of the first and second graphic objects is displayed on the keypad, based on application of a predetermined-type touch to the one graphic object, the controller changes one graphic object to the other graphic object.

In the mobile terminal, when the one graphic object is the first graphic object, the other graphic object may be the second graphic object, and when the one graphic object is the second graphic object, the other graphic object may be the first graphic object.

In the mobile terminal, the touch screen may include at least one or more input fields, and based on application of a touch to one of the at least one or more input field, the controller may output the keypad that includes at least one of the first and second graphic objects.

In the mobile terminal, the graphic object that differs according to a type of information in conjunction with the input field to which the touch is applied, which is able to be input, may be output to the keypad.

In the mobile terminal, when the input field into which text that is a type of information is able to be input is touched on, the first graphic object may be displayed on the keypad, and, when the input field into which an image that is a type of information is able to be input is touched on, the second graphic object may be displayed on the keypad.

In the mobile terminal, when one of the first and second graphic objects is selected, the controller may output an image that is received through the camera, to the touch screen.

In the mobile terminal, when the first graphic object is selected, the controller may output the image to one region of the touch screen, and, when the second graphic object is selected, may output the image to all regions of the touch screen.

In the mobile terminal, the one region may differ according to a type of touch that is applied to the first graphic object.

In the mobile terminal, when a first type touch is applied to the first graphic object, the controller may output the image on an already-selected input field, and, when a second type touch that is different from the first type touch is applied to the first graphic object, may output the image to at least one portion of the keypad.

In the mobile terminal, in a state where the image is output to the touch screen based on the selection of the first graphic object, when a drag touch is applied to screen information that corresponds to text which is included in the image, the controller may select the screen information to which the drag touch is applied.

In the mobile terminal, based on a user request, the controller may convert the selected screen information into text, and inputs the text that results from the conversion, into an already-selected input field.

In the mobile terminal, the screen information may be selected based on the drag touch that is applied in a first direction, and, when the drag touch is applied, in a second direction perpendicular to the first direction, to the selected screen information, the controller may selection-cancel at least one portion of the selected screen information.

In the mobile terminal, the controller may convert all pieces of screen information that correspond to the text which is included in the image, into text, based on a user request, may select the screen information based on a drag touch that is applied in a first direction, and, when the drag touch is applied, in a third direction that is opposite to the first direction, to the screen information that is included in the image, and then the user request is made, the controller may convert portions other than a portion to which the drag touch is applied in the third direction, of the screen information that corresponds to the text which is included in the image, into text.

In the mobile terminal, in a state where the image is output to the touch screen based on the selection of the second graphic object, based on application of a touch to a photographing button, the controller may capture the image and display the captured image on an already-selection input field.

In the mobile terminal, when a drag touch is applied in one direction to the image and then the touch is applied to the photographing button, the controller may capture the image in such a manner that an image that corresponds to a region to which the drag touch is applied is not output.

DETAILED DESCRIPTION OF THE DISCLOSURE

Reference is now made toFIGS. 1A to 1C, whereFIG. 1Ais a block diagram of a mobile terminal in accordance with the present disclosure, andFIGS. 1B and 1Care conceptual views of one example of the mobile terminal, viewed from different directions.

The input unit120includes a camera121for obtaining images or video, a microphone121, which is one type of audio input device for inputting an audio signal, and a user input unit122(for example, a touch key, a push key, a mechanical key, a soft key, and the like) for allowing a user to input information. Data (for example, audio, video, image, and the like) is obtained by the input unit120and may be analyzed and processed by controller180according to device parameters, user commands, and combinations thereof.

The sensing unit140is typically implemented using one or more sensors configured to sense internal information of the mobile terminal, the surrounding environment of the mobile terminal, user information, and the like. For example, inFIG. 1A, the sensing unit140is shown having a proximity sensor141and an illumination sensor142.

The display unit151may have an inter-layered structure or an integrated structure with a touch sensor in order to facilitate a touch screen. The touch screen may provide an output interface between the mobile terminal100and a user, as well as function as the user input unit122which provides an input interface between the mobile terminal100and the user.

At least some of the above components may operate in a cooperating manner, so as to implement an operation or a control method for a glass type terminal according to various embodiments to be explained later. The operation or the control method for the glass type terminal may be implemented on the glass type terminal by driving at least one application program stored in the memory170.

The mobile communication module112can transmit and/or receive wireless signals to and from one or more network entities. Typical examples of a network entity include a base station, an external mobile terminal, a server, and the like.

Such network entities form part of a mobile communication network, which is constructed according to technical standards or communication methods for mobile communications (for example, Global System for Mobile Communication (GSM), Code Division Multi Access (CDMA), CDMA2000 (Code Division Multi Access 2000), EV-DO (Enhanced Voice-Data Optimized or Enhanced Voice-Data Only), Wideband CDMA (WCDMA), High Speed Downlink Packet access (HSDPA), HSUPA (High Speed Uplink Packet Access), Long Term Evolution (LTE), LTE-A (Long Term Evolution-Advanced), and the like). Examples of wireless signals transmitted and/or received via the mobile communication module112include audio call signals, video (telephony) call signals, or various formats of data to support communication of text and multimedia messages.

A mobile terminal according to one embodiment of the present invention, which is capable of including at least one among the constituent elements described above, outputs a keypad.

The keypad is a means of enabling text, an emoticon, an image, and the like to be input into an input field (an input region, an input window, or the like) that is included in a touch screen151, and is understood as a keypad, a keyboard, or the like that is realized in software. The keypad is referred to as keypad screen information, a keyboard, keyboard screen information, or the like.

The input field means a filed (or region) for inputting information from a user, of screen information that is output to the touch screen151. For example, the input field means a region of screen information relating to a memo pad, into which information (data) (for example, an image, a file, text, or the like) is input by the user, a region of screen information relating to a schedule, in which information is input by the user, or the like.

As one example, the keypad is output to the touch screen151based on application of a touch to the input field (based on selection made on the input field or on activation performed on the input field).

On the other hand, the mobile terminal according to the present invention performs an OCR function using an image that is received through a camera121. The OCR function means a function of converting screen information (an image) corresponding to text that is included in the image into text.

A controller180displays text that results from performing the OCR function, on in an input field that is displayed on the touch screen151. For example, the controller180inputs text that results from performing the OCR, to the input field of the screen information that is output to the touch screen151. As another example, after the text results from performing the OCR function, when the touch is applied to the output field into which the text that results from the conversion is to be input, the controller180inputs the text that results from the conversion, into the input field to which the touch is applied. As still another example, after at least one portion of the screen information that corresponds to the text that is included in the image is selected through the OCR, when a drag touch that starts from at least the portion is released from the input field that is output to the touch screen151, the controller180may input the text that corresponds to the screen information, into the input field.

On the other hand, the mobile terminal according to the present invention performs a photographing function using the image that is received through the camera121. The photographing function means a function of capturing the image that is received through the camera121. The captured image is stored in a memory170or is displayed on the already-selected input field.

Referring to the accompanying drawings, a method will be described in detail below in which it is possible to input information in an optimized way using the keypad, the input field, the OCR function, and the photographing function.

FIG. 2is a flowchart for describing a typical control method according to the present invention.FIGS. 3A and 3Bare a conceptual diagram for describing the control method that is described referring toFIG. 2.

First, referring toFIG. 2, according to the present invention, a step of receiving a keypad output request is performed (S210). Specifically, the controller180receives the keypad output request in various ways.

For example, when a touch is applied to any one of the input fields that are included in the touch screen151, the controller180determines the touch that is applied to the touch field, as the keypad output request. Based on the application of the touch on the input field, the controller180displays a keypad200on at least one portion of an output region of the touch screen151.

As another example, in a case where an icon in conjunction with a keypad output function is displayed on the touch screen151, based on selection of the icon, the controller180displays the keypad200on at least on portion of the output region of the touch screen151.

The keypad200that is output disappears from the touch screen151in various ways. For example, in a state where the keypad200is output, if the touch is applied to a region other than the region to which the keypad200is output, the controller180causes the keypad200to disappear.

As another example, the keypad200includes a button (an icon or a key) in conjunction with a function of causing the keypad200to disappear. In this case, based on selection of the button, the controller180causes the keypad200that is output, to disappear from the touch screen151.

Thereafter, according to the present invention, based on the keypad output request, a step of outputting the keypad200including a first graphic object in conjunction with the OCR function relating to the camera121and a second graphic object in conjunction with the photographing function relating to the camera121is performed (S220).

Specifically, based on the keypad output request, the controller180outputs the keypad200that includes at least one of a first graphic object300ain conjunction with the optical character reader (OCR) function relating to the camera121and a second graphic300bin conjunction with the photographing function relating to the camera121, to the touch screen151.

As one example,FIG. 3Aillustrates that an execution screen of an application relating to a message is output to the touch screen151. The execution screen includes an input field210for inputting information (text, an image, a moving image, or an emoticon). At this point, the input field210may be understood as a region on which the information that is input by the user is displayed.

Referring toFIG. 3A, based on the application of the touch to the input field210that is included in the touch screen151, the controller180outputs the keypad200to at least one portion of the output region of the touch screen151.

At this time, as illustrated inFIG. 3A, the first and second graphic objects300aand300bare together displayed on the keypad200.

On the other hand, as illustrated in part (a) ofFIG. 3B, only one graphic object (for example, the first graphic object300a) of the first and second graphic objects300aand300bmay be displayed on the keypad200.

Whether the first and second graphic objects300aand300bare together displayed on the keypad200or only one graphic object of the first and second graphic objects300aand300bis displayed on the keypad200is determined by a user's setting or a controller's control.

At this point, as one example, based on a type of information in conjunction with the input field that is included in the touch screen151, which can be input, the controller180determines whether the first and second graphic objects300aand300bare together displayed on the keypad200or only one graphic object of the first and second graphic objects300aand300bis displayed on the keypad200.

For example, in a case where the type of information that can be input into the input field include both of text and an image, the first and second graphic objects300aand300bare all displayed on the keypad200.

As another example, in a case where the type of information that can be input into the input field is text, the controller180output only the first graphic object300ain conjunction with the OCR function to the keypad200. As another example, in a case where the type of information that can be input into the input field is an image, only the second graphic object300bin conjunction with the photographing function is output to the keypad200.

Additionally, the controller180outputs the first and second graphic objects300aand300bin various ways. Contents relating to this will be described in more detail below referring toFIG. 5andFIGS. 6A to 6D.

Referring back toFIG. 3B, in a case where one graphic object of the first and second graphic objects300aand300bis displayed on the keypad200, based on application of a predetermined-type to the one graphic object, the controller180changes the one graphic object to the other graphic object.

At this point, the predetermined-type touch is a touch in conjunction with a function of changing the graphic object that is displayed on the keypad200. As the predetermined-type touch, there are various types of touches.

For example, the various types of touches include a short (or tap) touch, a long touch, a double touch, a multi touch, a drag touch, a flick touch, a pinch-in touch, a pinch-out touch, a swipe touch, a hovering touch, and the like.

The various types of touches will be described in more detail below.

The short (or tap) touch is a touch in which a touch object (for example, a finger or a stylus pen) is brought into contact with the touch screen151and is separated from the touch screen151within a given time thereafter (which is released within a given time after being applied to the touch screen151). For example, the short (or tap) touch is a touch in which, like a single click with a mouse, the touch object is kept into contact with the touch screen151for a short time.

The long touch is a touch in which the touch object is brought into contact with the touch screen151and stays there for a given time thereafter. For example, the long touch is a touch that is applied to the touch screen151using the touch object and is maintained for a given time thereafter. More specifically, the long touch is a touch that is released after being maintained for a given time at one point on the touch screen151. In addition, the long touch may be understood as a touch that corresponds to a touch and hold operation in which the touch object is kept into contact with the touch screen151for a given time or more.

The double touch refers to the short touch that is applied consecutively at least two times to the touch screen151for a given time.

The given time that is used when describing the short touch, the long touch, and the double touch is determined by a user's setting.

The multi touch is a touch in which is applied to at least two contact points on the touch screen151substantially at the same time.

The drag touch is a touch that is applied to a first point on the touch screen151and is slid along one direction to a second point on the touch screen151, which is different from the first point, and is released from the second point.

Specifically, the drag touch is a touch in which the touch object is brought into contact with a first point on the touch screen151, is dragged along one direction to a second point on the touch screen151, which is different from the first point, and is separated from the second point.

In addition, the drag touch may mean a touch that is applied to one point on the touch screen151and then is continuously slid from the one point.

The flick touch refers to the drag touch that is applied within a given time. Specifically, the flick touch is a touch in which the touch object that is brought into contact with one point on the touch screen151is separated from the touch screen151within a given time. In other words, the flick touch is understood as the drag touch that is applied at a predetermined speed or above.

The swype touch is the drag touch that is applied in a straight line.

The pinch-in touch is first and second touches which are applied to two different points (two separately-positioned points) on the touch screen151, respectively, and at least one of which is slid toward the other. For example, the pinch-in touch is a touch that is realized by an operation in which two fingers are brought into contact with two separately-positioned points on the touch screen151, respectively, and in such a state, a distance between the two fingers is decreased.

The pinch-out touch is first and second touches which are applied to two different points (two separately-positioned points) on the touch screen151, respectively, and at least one of which is slid away from the other. For example, the pinch-out touch is a touch that corresponds to an operation in which two fingers are brought into contact with two separately-positioned points on the touch screen151, respectively, and in such a state, a distance between the two fingers is increased (the two fingers are moved away from each other).

The hovering touch is a touch that corresponds to an operation in which the touch object is moved in space in a state of being separated from the touch screen151without coming into contact with the touch screen151. One example of the hover touch is a proximity touch that is described referring toFIG. 1A. For example, the hovering touch is a touch that corresponds to an operation in which the touch object stays at a point in space for a given time or above in a state of being positioned separately from the touch screen151.

In the present specification, the long touch is described, for example, as a predetermined-type touch in conjunction with a function of changing a graphic object being displayed on the keypad200to a different graphic object is an example of the long touch.

In a case where only one graphic object (for example, the graphic object300a) of the first and second graphic objects300aand300b, as illustrated in part (a) ofFIG. 3B, is displayed on the keypad200, the controller180, as illustrated in part (b) ofFIG. 3B, detects that a predetermined-type touch (the long touch) is applied to the one graphic object300a.

Based on the application of the predetermined-type touch (the long touch), the controller180, as illustrated in part (c) ofFIG. 3B, changes the one graphic object300ato the other graphic object300b.

At this time, when the one graphic object is the first graphic object300a, the other graphic object is the second graphic object300b. When the one graphic object is the second graphic object300b, the other graphic object is the first graphic object300a. As one example, this operation is understood as a toggle operation.

With this configuration, according to the present invention, the keypad200is provided that enables the OCR function or the photographing function to be selected in a more convenient manner.

Referring to the accompanying drawings, a method will be described in more detail below in which various pieces of information are input using the first graphic object300ain conjunction with the OCR function and the second graphic object300bin conjunction with the photographing function.

FIGS. 4A and 4Bare conceptual diagrams for describing the method of inputting information using the OCR function and the photographing function according to one embodiment of the present invention.

When one of the first graphic object300ain conjunction with the OCR function and the second graphic object300bin conjunction with the photographing function is selected (touched on), the controller180outputs an image320that is received through the camera121.

The image320that is received through the camera121is referred to as a preview image. The preview image means an image that is received through the camera121and is displayed in real time. Screen information (an image or the like) that is included in the preview image is changed based on movement of a photographic subject or a mobile terminal100.

On the other hand, when the first graphic object300ain conjunction with the OCR function is selected, the controller180outputs the image320to one region of the touch screen151. When the second object300bin conjunction with the photographing function is selected, the controller180outputs the image320to all output regions of the touch screen151.

The outputs region of the touch screen151may correspond to all regions to which the screen information can be output, and may mean regions other than a status bar, among all the regions.

When the first graphic object300ain conjunction with the OCR function relating to the camera121, as illustrated in part (a) ofFIG. 4A, is selected, the controller180, as illustrated in part (b) ofFIG. 4A, outputs the image320that is received through the camera121, to one region of the touch screen151. The one region differs according to a type of touch that is applied to the first graphic object300a.

Contents relating to this will be described in more detail below referring toFIG. 7andFIGS. 8A and 8B.

When the second graphic object300bin conjunction with the photographing function relating to the camera121, as illustrated in part (a) ofFIG. 4B, is selected, the controller180, as illustrated in part (b) ofFIG. 4B, outputs the image320that is received through the camera121, to all output regions of the touch screen151.

However, limitation to this is not imposed. Although the first graphic object300ain conjunction with the OCR function is selected, the controller180controls the touch screen151in such a manner that the image320that is received through the camera121is output to all the output regions of the touch screen151.

Referring back toFIG. 4A, the image320that is received through the camera121, as illustrated in part (b) ofFIG. 4A, includes screen information (an image) that corresponds to text, when the camera121faces a photographic subject402on which the text is printed.

At this time, when a predetermined-type touch (for example, the drag touch), as illustrated in part (b) ofFIG. 4A, is applied to screen information340that corresponds to text which is included in the image320that is output to the touch screen151, the controller180, illustrated in part (c) ofFIG. 4A, converts the screen information340to which the drag touch is applied, into text410.

Specifically, based on the application of the predetermined-type touch (the drag touch) on the image320, the controller180selects screen information that is included in the image320. At this time the selected screen information is screen information that corresponds to a path that is traced by the predetermined-type touch (the drag touch) that is applied.

In a case where screen information that corresponds to text is included in the selected screen information, based on selection of button330in conjunction with the OCR function (that is, a function of converting an image into text or a function of performing conversion into text), the controller180converts the screen information into text.

Thereafter, the controller180inputs (displays) the text410that results from the conversion, into (on) an input field400(for example, an already-selected input field).

On the other hand, as illustrated in part (b) ofFIG. 4B, in a case where the graphic object300bin conjunction with the photographing function is selected, the image320that is received through the camera121and a photographing button360that performs an photographing operation are displayed on the touch screen151.

When the photographing button360is selected, the controller180captures the image320that is received through the camera121, and as illustrated in part (c) ofFIG. 4B, displays a captured image420on an input field400(for example, an already-selected input field).

InFIG. 4A, a type of information that is input into the input field is text. InFIG. 4B, a type of information that is input into the input field is an image.

With this configuration, according to the present invention, in a case where information is input into the input field, a method and an UI/UX are provided in which text or an image is selected in a more convenient manner and is input (applied) into (to) the input field.

Referring to the accompanying drawings, a method will be described in more detail below in which one graphic object of the first and second graphic objects300aand300bthat are described referring toFIG. 3, is output to the keypad200.

FIG. 5is a flowchart for describing a method of outputting the first graphic object300ain conjunction with the OCR function and the second graphic object300bin conjunction with the photographing function to the keypad200according to one embodiment of the present invention.FIGS. 6A to 6Dare conceptual diagrams for describing the method that is described referring toFIG. 5.

Referring toFIG. 5, according to the present invention, a step of selecting the input field is performed (S510).

Referring toFIG. 6A(a), the touch screen151includes at least one input field600aor600b.

When a touch is applied to at least one input field of at least one or more input fields that are included in the touch screen151, the controller180selects an input field to which the touch is applied.

At this time, the selected input field has a different form than a pre-selection input field. For example, an indicator indicating that the input field is selected is displayed on the selected input field. In addition, a cursor is displayed on the selected input field. In addition, in a state where a predetermined alerting information is displayed before an input field is selected, when a touch is applied to the input field, the predetermined alerting information is not displayed or is displayed in a dimmer (more lightened) manner on the selected input field.

Based on the application of the touch to one of at least one or more input fields, the controller180outputs the keypad200that includes at least one of the first graphic object300ain conjunction with the OCR function and the second graphic object300bin conjunction with the photographing function, to the touch screen151.

At this time, the graphic object that differs according to a type of information in conjunction with the input field to which the touch is applied, which can be input, is output to the keypad.

To this end, the controller180determines whether or not an attribute (for example, a type of information that can be input) in conjunction with the selected input field is text or an image (S520).

In a case where the attribute (the type of information that can be input) is text, the controller180outputs the keypad200that includes the first graphic object300ain conjunction with the OCR function, to the touch screen151(S530).

On the other hand, in a case where the attribute (the type of information that can be input) is an image, the controller180outputs the keypad200that includes the graphic object300bin conjunction with the photographing function, to the touch screen151(S540).

At this time, the keypad200includes only the first graphic object300aor only second graphic object300b.

For example, as illustrated in part (a) ofFIG. 6A, the touch screen151includes at least one input field500aor500b. At this time, one of at least one or more input fields is an input field500ainto which text that is a type of information can be input, and the other is an input field500binto which an image that is a type of information can be input.

When the input field500ainto which text that is a type of information can be input, as illustrated in part (a) ofFIG. 6A, is touched on, the controller180, as illustrated in part (b) ofFIG. 6A, outputs the keypad200that includes the first graphic object300ain conjunction with the OCR function, to the touch screen151. That is, based on the touch-on of the input field500ainto which text that is a type of information can be input, the keypad200includes the first graphic object300ain conjunction with the OCR function.

When the input field500binto which an image that is a type of information can be input, as illustrated in part (a) ofFIG. 6B, is touched on, the controller180, as illustrated in part (b) ofFIG. 6B, outputs the keypad200that includes the second graphic object300bin conjunction with the photographing function, to the touch screen151. That is, based on the touch-on of the input field500binto which an image that is a type of information can be input, the keypad200includes the second graphic object300bin conjunction with the photographing function.

In a state where the keypad200including the second graphic object300bis output, when a touch, as illustrated in part (b) ofFIG. 6B, is applied to the input field500ainto which text that is a type of information can be input, the controller180, as illustrated in part (c) ofFIG. 6B, changes the second graphic object300bthat is displayed on the keypad200, to the first graphic object300a.

Although not illustrated, in a state where the keypad200on which the first graphic object300ais displayed is displayed on the touch screen151, a touch is applied to the input field500binto which text that is a type of information can be input, the controller180changes the graphic object300athat is displayed on the keypad200, to the second graphic object300b.

On the other hand, the present invention is not limited to the embodiments described above. For example, the controller180may display at least one of the first graphic object300aand the second graphic object300bon a region other than the keypad200.

For example, as illustrated inFIG. 60, the first graphic object300ain conjunction with the OCR function is displayed on the keypad200, the second graphic object300bin conjunction with the photographing function is displayed on the vicinity of an input field500cinto which an image that is a type of information can be input.

Although not illustrated, the reverse of this is possible. That is, the second graphic object300bin conjunction with the photographing function may be displayed on the keypad200, and the first graphic object300amay be displayed on the vicinity of the input field into which text that is a type of information can be input.

Based on the selection of the input field, at least one of the first graphic object300aand the second graphic object300bis displayed on the vicinity of the input field. That is, at least one of the first graphic object300aand the second graphic object300bmay be displayed by default on the vicinity of the input field, and, based on the selection (the touch-on) of the input field, the input field may be output to the vicinity of the input field.

For example, in a state where the first graphic object300aand the second graphic object300bare not output, when the input field into which text that is a type of information can be input, as illustrated in part (a) ofFIG. 6D, is selected, in response to the selection, the controller180outputs the first graphic object300ain conjunction with the OCR function to the vicinity of the selected input field.

For example, in the state where the first graphic object300aand the second graphic object300bare not output, when the input field into which an image that is a type of information can be input, as illustrated in part (b) ofFIG. 6D, is selected, in response to the selection, the controller180outputs the second graphic object300bin conjunction with the photographing function to the vicinity of the selected input field.

With this configuration, according to the present invention, a user interface can be provided through which the graphic object in conjunction with the OCR function or the photographing function is output in an optimized way according to the input field into which the user wants to input text or an image.

A method of performing the OCR function or the photographing function described above will be described in more detail referring to the accompanying drawings.

FIG. 7is a flowchart for describing a method of outputting the image that is received through the camera121and a method of inputting text using the image that is output, according to one embodiment of the present invention.FIGS. 8A and 8Bare conceptual diagrams for describing the methods that are described referring toFIG. 7.

Referring toFIG. 7, as described above, based on the selection of the input field that is displayed on the touch screen151, the controller180outputs the keypad200that includes at least one of the first graphic object300ain conjunction with the OCR function and the second graphic object300bin conjunction with the photographing function, to the touch screen151(S710).

When at least one of the first and second graphic objects300aand300bis selected, the controller180outputs an image that is received through the camera121, to the touch screen151.

At this time, when the first graphic object300ais selected, the controller180outputs the image to one region of the touch screen151. When the second graphic object300bis selected, the controller180outputs the image to all regions of the touch screen151.

At this time, the controller180outputs the image to different regions of the touch screen151according to a type of touch that is applied to the first graphic object300a. That is, the one region differs according to the type of touch that is applied to the first graphic object300a.

To this end, according to the present invention, a step of determining whether a first type touch or a second type touch is applied to the first graphic object300ain conjunction with the OCR function that is included in the keypad200is performed (S720). Specifically the first type touch and the second type touch are among the various types of touches that are described above. The first type touch and the second type touch are different from each other.

When the result of the determination is that a touch type that is applied to the first graphic object300ais the first type touch, the controller180outputs an image (a preview image) that are received through the camera121, to the input field that is already elected in S710(S730).

When the result of the determination is that a touch type that is applied to the first graphic object300ais the second type touch that is different from the first type, the controller180outputs the image (the preview image) that is received through the camera121, to at least one portion of the region on which the keypad200is displayed, of the touch screen151(S740).

Subsequently, according to the present invention, a step of selecting screen information that corresponds to text that is included in the image (the preview image) is performed (S750). For example, in a state where the image is displayed on the already-selected input field, when a touch is applied to the image in a state where screen information corresponding to text is displayed on the image, the controller180selects the screen information that is displayed on the image.

As another example, based on the application of the drag touch to the image, the controller180selects at least one portion of the screen information that corresponds to the text which is included in the image.

Subsequently, the controller180converts the selected screen information into text (S760). The conversion into text is performed based on a user request (for example, selection of a button in conjunction with the function of performing the conversion into text), or is performed based on selection of the screen information that corresponds to the text in the image.

Subsequently, the controller180inputs the text that results from the conversion to the input field that is selected in S710(S770).

The contents that are described referring toFIG. 7can be understood more apparently fromFIGS. 8A and 8B.

As illustrated in part (a) ofFIG. 8A, based on selection of an input field600, the keypad200that includes the first graphic object300ain conjunction with the OCR function is displayed on the touch screen151. When the first type touch (for example, the short touch) is applied to the first graphic object300a, the controller180, as illustrated in part (b) ofFIG. 8A, outputs an image (a preview image) that is received through the camera121, to the already-selected input field600.

Screen information630athat corresponds to text that is printed on a photographic subject620is display on the image.

As illustrated in part (c) ofFIG. 8A, in a state where screen information (the image)630athat correspond to the text is included in an image that is displayed on the input field600, based on the application of the image, the controller180selects the screen information630a.

Subsequently, the controller180converts the selected screen information630ainto text, and inputs text630athat results from the conversion, into the already-selected input field600.

On the other hand, when the second type touch (for example, the long touch) that is different from the first type touch, as illustrated in part (a) ofFIG. 8B, is applied to the first graphic object300athat is included in the keypad200, the controller180, as illustrated in part (b) ofFIG. 8B, outputs an image610that is received through the camera121, to at least one portion of an region to which the keypad200is output.

At this time, the controller180, as illustrated in part (b) ofFIG. 8B, may display the image (the preview image)610that is received through the camera121, on all regions (a keypad region) to which the keypad200is output, and, as illustrated inFIG. 4Amay display the image320on one portion of the region to which the keypad200is output.

Likewise, the screen information630athat corresponds to the text that is printed on the photographic subject620is included in the image610.

At this time, as illustrated in part (c) ofFIG. 8A, when the drag touch is applied to at least one portion of the screen information630athat corresponds to the text which is included in the image610, the controller180selects screen information640to which the drag touch is applied.

In this case, a graphic effect is changed or an indicator is displayed on the selected screen information640(or the path that is traced by the drag touch) in order to alert the user that the screen information640is selected.

Subsequently, based on the user request (for example, the selection of the button in conjunction with the function of performing the conversion into text, or a completion button), the controller180converts the selected screen information640into text. Subsequently, as illustrated in part (d) ofFIG. 8B, the controller180inputs text640bthat results from the conversion, into the already-selected input field600.

With this configuration, according to the present invention, a user interface is provided through which, when performing the OCR function, an image that is received through the camera121is displayed on the touch screen in various ways, and at least one portion of the screen information that corresponds to the text which is included in the image is converted into text.

On the other hand, according to the present invention, an interface is provided through which, based on the drag touch, only a portion that the user wants, of the screen information that is included in the image is selected and is converted into text.

FIG. 9is a flowchart for describing a method of converting the screen information corresponding to the text that is included in the image which is received through the camera121, according to one embodiment of the present invention.FIGS. 10A, 10B, 11A, and 11Bare conceptual diagrams for describing the method that is described referring toFIG. 9.

Based on the selection of the first graphic object300ain conjunction with the OCR function, the controller180outputs an image that is received through the camera121, to the touch screen151.

At this time, in a state where the image is output to the touch screen151, when the drag touch is applied to screen information that corresponds to text which is included in the image, the controller180selects the screen information to which the drag touch is applied.

At this time, based on the drag touch that is applied in a first direction, the screen information is selected.

Specifically, based on the drag touch that is applied in the first direction to an image (a preview image) that is received through the camera121, the controller180selects at least one portion of the screen information that corresponds to the text which is include in the image (S910).

On the other hand, in a state where at least the one portion of the screen information is selected, when the drag touch is applied, in a second direction different from (perpendicular to) the first direction, to the selected screen information, the controller180selection-cancels at least one portion of the selected screen information (S920).

Specifically, in a state where at least one portion of the screen information that is included in the image is selected based on the drag touch that is applied in the first direction, when the drag touch is applied in the second direction different from the first direction, the controller180selection-cancels a portion of the selected screen information, which overlaps a path that is traced by the drag touch which is applied in the second direction.

Subsequently, based on the user request (for example, the selection of the button in conjunction with the function of performing the conversion into text, or the completion button), the controller180converts the selected screen information into text, and inputs the text that results from the conversion into the input field (for example, an already-selected input field) (S930).

Referring toFIG. 10A, based on the selection of the graphic object in conjunction with the OCR function, an image that is received through the camera121is displayed on the touch screen151. In this state, when a drag touch T1(for example, a drag touch that is applied to the right), as illustrated in part (a) ofFIG. 10A, is applied in the first direction to the screen information630athat corresponds to the text which is included in the image (the preview image), the controller180selects screen information640ato which the drag touch T1is applied.

At this time, the controller180changes a graphic effect of the selected screen information in the mage or displays the selected screen information in such a manner that the selected screen information is visually distinguishable from the other screen information (image), in order to alert the user that the screen information640ais selected.

In a state where the screen information is selected, when a drag touch T2, as illustrated in part (b) ofFIG. 10A, is applied in the second direction (for example, in the downward direction) perpendicular to the first direction, the controller180selection-cancels at least one portion of the selected screen information.

As illustrated in part (b) ofFIG. 10A, at least the one portion that is selection-canceled is a portion in which the selected screen information640aand the drag touch T2that is applied in the second direction overlaps each other.

Accordingly, the drag touch T2is applied in the second direction, the selected second direction640ais divided into at least two or more pieces of screen information642a,642b, and643c.

Subsequently, based on the user request (the selection of the button in conjunction with the function of performing the conversion into text or the completion button), the controller180, as illustrated in part (c) ofFIG. 10A, converts the selected pieces of screen information642a,642b, and642cinto text and inputs a result of the conversion into an input field600a.

On the other hand, in a state where screen information that corresponds to text is included in an image that is received through the camera121, when the button (or the completion button) in conjunction with the function of performing the conversion into text is selected without a separate operation of selecting at least one portion of the screen information, the controller180converts all pieces of screen information that are included in the image, into text.

That is, based on the user request, the controller180converts all pieces of screen information that correspond to the text which is included in the image, into text.

On the other hand, when the drag touch is applied in the first direction to the image, based on the drag touch, the controller180selects at least one portion of the screen information.

After the drag touch is applied, in a third direction (in the leftward direction) that is opposite to the first direction (for example, in the rightward direction), to the screen information that is included in the image, when the user request (the selection of the button in conjunction with the function of performing the conversion into text, or the completion button) is made, the controller180converts screen information other than the screen information to which the drag touch is applied in the third direction, of the screen information that corresponds to the text which is included in the image.

For example, in a state where the preview image is output to the touch screen151according to the selection of the first graphic object300a, when the completion button, as illustrated in part (a) ofFIG. 10B, is selected without applying a separate touch, the controller180converts all pieces of screen information that correspond to the text which is in the preview image, into text.

On the other hand, when a drag touch T3is applied, in the third direction that is opposite to the first direction, to the image610, the controller180, as illustrated in part (b) ofFIG. 10B, performs blinding processing on the screen information to which the drag touch T3is applied, of the screen information that corresponds to the text which is include in the preview image.

In this state, when the user request is received (the button (the completion button) in conjunction with the function of performing the conversion into text is selected), the controller180, as illustrated in part (c) ofFIG. 10B, converts only screen information other than the screen information to which the drag touch T3is applied, into text. The text that results from the conversion is input into the input field.

With this configuration, according to the present invention, an optimized UI/UX is provided through which only one portion that is included in the image that is received through the camera121can be converted into text or portions other than the one portion can be all converted into text.

An additional embodiment of the OCR function (a method of performing the conversion into text) according to the present invention will be described in more detail below referring to the accompanying drawings.

As illustrated in part (a) ofFIG. 11A, a button1110in conjunction with a function of inputting the screen information into the preview image is displayed on the touch screen151. The button1110is displayed on a preview image1100.

After the button1110, as illustrated in part (a) ofFIG. 11A, is selected, when the drag touch, as illustrated in part (b) ofFIG. 11A, is applied to the preview image1100, the controller180displays screen information1130athat correspond to a path that is traced by the drag touch, on the preview image1110.

In addition, when the drag touch is applied in the first direction to the screen information that corresponds to the text which is included in the preview image1110, the controller180selects screen information1120to which the drag touch is applied.

In a state where the screen information1120is selected or the screen information1130athat corresponds to the path which is traced by the drag touch is displayed on the preview image1100, when the button (or the completion button) in conjunction with the function of performing the conversion into text is selected, the controller180converts the selected screen information1120into text1122bor converts the screen information1130athat corresponds to the path which is traced by the drag touch, into text1130b.

Subsequently, the items of text1122band1130bthat result from the conversion are displayed on an input field1150that is already selected. At this point, based on positions of the selected screen information1120and the screen information1130athat corresponds to the path which is traced by the drag touch, the controller180inputs the items of text that result from the conversion, into the already-selected input field1150.

With this configuration, according to the present invention, a user interface is provided through which not only the screen information that corresponds to the text which is included in the image that is received through the camera121, but also the screen information that is input additionally by the user is converted into text at a time.

On the other hand, in a state where, screen information1122athat corresponds to the text, of the information that is included in the preview image1100, as illustrated in part (a) ofFIG. 11B, is screen information, based on application of a predetermined-type touch74, the controller180selects at least one portion of the screen information1122a.

The predetermined-type touch74is a touch that is applied to the selected screen information1122a. The predetermined-type touches74include the drag touch, the long touch, the double touch, the drag touch, and the like that are applied in such a manner as to form a looped curve.

A visual effect is applied to, or an indicator indicates at least one portion1142aof the screen information to which the predetermined-type touch74is applied, of the selected screen information1122ain such a manner that at least the one portion1142ais distinguishable from the other portions of the screen information.

On the other hand, when at least one portion1142aof the screen information is selected, the controller180displays buttons1140in conjunction with a graphic effect that is to be applied to the text that results from the conversion, on the touch screen151. As one example, the buttons1140are displayed on the preview image1100.

In a state where at least one portion1142af the screen information is selected, after at least one button1142(for example, a button in conjunction with a boldface function of making characters thicker) is selected, when the button (or the completion button) in conjunction with the function of performing the conversion into text is selected, the controller180, as illustrated in part (c) ofFIG. 11B, converts the selected screen information1122ainto text, and inputs (displays) text that results from the conversion, into (on) the input field1150.

At this time, text1124bthat corresponds to at least one portion1142aof the screen information is displayed on the input field1150in a state where a graphic effect in conjunction with at least one button1142is applied to the text1142b.

In a case where at least the selected one button1142, as illustrated in part (c) ofFIG. 11B, is a button in conjunction with the boldface function of making characters thicker, text that corresponds to at least one portion1142aof the screen information is input into an input field151in a state of being boldfaced.

The example is describe above in which the screen information that is included in the preview image is selected and then at least one portion of the screen information is selected, but the present invention is not limited to this. That is, based on the application of the drag touch to the screen information being included in the preview image in the first direction, the controller180selects the screen information and outputs the buttons1140in conjunction with the graphic effect that is to be applied to the text that results from the conversion, to the touch screen151. After at least one of the buttons1140is selected, when the button (the completion button) in conjunction with the function of performing the conversion into text is selected, the controller180converts the selected screen information into text, and inputs the text that results from the conversion, into the input field in a state where the graphic effect in conjunction with the selected button is applied to the text that results from the conversion.

With this configuration, according to the present invention, a user interface is provided through which the graphic effect that is to be applied to the text that results from the conversion is selected in advance in a state where the preview image is output. Thus, a cumbersome procedure for applying the graphic effect to the text that results from the conversion is omitted, thereby remarkably improving user convenience.

A method of inputting an image in the input field using the second graphic object300bin conjunction with the photographing function will be described in more detail below referring to the accompanying drawings.

FIGS. 12A and 12Bare conceptual diagrams for describing the method of inputting the captures image in the input field in a case where the second graphic object300bin conjunction with the photographing function is selected.

As illustrated in part (a) ofFIG. 12A, the second graphic object300bin conjunction with the photographing function is displayed on the touch screen151. The second graphic object300bis included in the keypad200or is displayed on the vicinity (or within) the input field.

In addition, the second graphic object300bis displayed by default on the vicinity of the input field, or is displayed based on the selection (the touch-on) of the input field.

When the second graphic object300bis selected, the controller180, as illustrated in part (b) ofFIG. 11A, outputs the image that is received through the camera121, to the touch screen151.

In a state where, based on the selection of the second graphic object300b, an image1210is output to the touch screen151, when a touch is applied to the photographing button360, the controller180captures the image1210and displays the captures image1220on an input field1200that is already selected.

On the other hand, after the drag touch is applied in one direction to the image1210, when a touch is applied to the photographing button360, the controller180captures the image in such a manner that an image that corresponds to a region to which the drag touch is applied is not output.

For example, in a state where, as illustrated in part (a) ofFIG. 12B, based on the application of the touch to the second graphic object300bin conjunction with the photographing function, the image1210that is received through the camera121is displayed on the touch screen151, a drag touch T4is applied in one direction (for example, to the leftward direction). In this case, the controller180, as illustrated in part (b) ofFIG. 12B, performs the blinding processing in such a manner that the image which corresponds to the region to which the drag touch is applied is not output.

In this state, when a touch is applied to the photographing button360, the controller180captures the image in such a manner that the image that corresponds to the region to which the drag touch is applied is not output, and, as illustrated in part (c) ofFIG. 12B, inputs the captured image (the image that is captured in a state where the image that corresponds to the region to which the drag touch is applied is not output)1240into the input field. Part (c) ofFIG. 12Billustrates one example in which the captured image1240is input into the input field and then the captured image1240is transmitted on an execution screen of a messenger-related application, through the use of a messenger.

With this configuration, according to the present invention, a user interface is provided through which an image as well as text can be input into the input field in a more convenient manner, and at least one portion of the image is input right into the input field in a state where the one portion of the image is not output.

On the other hand, according to the present invention, the OCR function and the photographing function are performed using user voice.

FIG. 13is a flowchart for describing a method of converting or capturing one portion of screen information that corresponds to text which is included in an image that is received through the camera121, according to one embodiment of the present invention.FIGS. 14A and 14Bare conceptual diagrams for describing the method that is described referring toFIG. 13.

Referring toFIG. 13, based on the selection of one of the first graphic object300aand the second graphic object300b, the controller180outputs the image that is received through the camera121, to the touch screen151(S1310).

In a state where the image is output, the controller180receives the user voice (S1320). Specifically, based on the output of the image, or based on selection of a microphone drive (activation) button that is displayed on the touch screen151, the controller180drives (activates) the microphone122and receives the user voice through the microphone122.

When the user voice is received, the controller180selects screen information that corresponds to voice information on the received user voice, of the screen information that corresponds to the text which is included in the preview image (S1330).

Subsequently, the controller180converts the selected screen information into text or captures only one portion of the preview image, which includes the selected screen information.

Specifically, in a case where the preview image is output based on the selection of the first graphic object300ain conjunction with the OCR function, the controller180converts the selected screen information into text, and inputs the text that results from the conversion, into the already-selected input field.

In addition, in a case where the preview image is output based on the selection of the second graphic object300bin conjunction with the photographing function, the controller180captures only one portion of the preview image, which includes the selected screen information, and inputs the captured one portion into the already-selected input field.

The contents described above can be understood more apparently fromFIGS. 14A and 14B.

When the first graphic object300ain conjunction with the OCR function, as illustrated in part (a) ofFIG. 14A, is selected, the controller180, as illustrated in part (b) ofFIG. 14A, outputs an image1410that is received through the camera121, to the touch screen151. The image1410is displayed on one region of the touch screen151.

A button1420in conjunction with a function of driving the microphone is displayed on the image1410. When the button1420is selected, the controller180, as illustrated in part (b) ofFIG. 0.14A, activates the microphone122. However, based on the selection of the first graphic object300a, or based on the displaying of the image1410on the touch screen151, the microphone122may be driven (activated).

In a case where the microphone122is activated, when user voice1430, as illustrated in part (c) ofFIG. 14A, is received, the controller180selects screen information1450athat corresponds to the user voice, of the screen information that corresponds to the text which is included in the image1410.

Subsequently, in a state where the screen information1450ais selected, when the button (or the completion button) in conjunction with the function of performing the conversion into text is selected, the controller180, as illustrated as in part (d) ofFIG. 14A, converts the selected screen information1450ainto text, and input text1450bthat results from the conversion, into the already-selected input field.

At this time, a type of information that is input into the input field is text.

On the other hand, as illustrated in part (a) ofFIG. 14B, based on the selection of the second graphic object300bin conjunction with the photographing function, the image that is received through the camera121is displayed on the touch screen151.

Likewise, the button1420in conjunction with the function of driving the microphone is displayed on the image1410. When the button1420is selected, the controller180, as illustrated in part (b) ofFIG. 14B, activates the microphone122. However, based on the selection of the second graphic object300b, or based on the displaying of the image1410on the touch screen151, the microphone122may be drive (activated).

In a state where the microphone122is activated, when user voice1450, as illustrated in part (c) ofFIG. 14B, is received, the controller180selects the screen information1450athat corresponds to the user voice, of the screen information that corresponds to the text which is included in the image1410.

Subsequently, in a state where the screen information1450ais selected, when the photographing button360is selected (touched on), the controller180, as illustrated in part (d) ofFIG. 14B, captures only one portion of the preview image, which includes the selected screen information.

Subsequently, the controller180inputs the captured image (the image that corresponds to one portion that includes the selected screen information)1450cinto the already-selected input field.

At this time, a type of information that is input into the input field is an image.

As described above, according to the present invention, an interface is provided through which information that the user wants can be input in an optimized way.

In addition, according to the present invention, a UI/UX is provided through which only a portion that corresponds to the screen information that the user wants, of the screen information that corresponds to the text which is included in the image that is received through the camera121is converted into text.

In addition, according to the present invention, at least one of the graphic object300ain conjunction with the OCR function and the graphic object300bin conjunction with the photographing function is output to the keypad200. Thus, an interface is provided through which information that the user wants to input is selected in a more convenient manner.