Patent Description:
Electronic devices (e.g., a mobile terminal, a smart phone, a wearable device, and the like) are capable of providing various functions. For example, an electronic device may provide functions, such as, for example, a voice communication function, a data communication function, a short-range wireless communication function (e.g., Bluetooth™, near field communication (NFC), or the like), a mobile communication function (e.g., <NUM>rd generation (<NUM>), <NUM>, <NUM>, or the like), a music or video reproduction function, a photographing or video recording function, a navigation function, or the like.

The publication <CIT> describes a mobile terminal including a camera, a display unit, and a controller. The controller, if a camera application is executed, is configured to output a camera preview and a general capturing button on the display unit, determine a capturing option based on information on a subject included in the camera preview, and control a variable capturing button to be outputted in a manner of being changed from the general capturing button based on the determined capturing options.

The publication <CIT> describes an identification method and an electronic device in which the identification method is implemented. This identification method comprises detecting an object using the electronic device, providing an identification box having an appearance which corresponds with the object as detected, and displaying the identification box having said appearance via the electronic device.

More particularly, an electronic device may provide a function of recognizing a document taken by a camera.

In accordance with the invention, an electronic device is provided according to the appended independent device claim.

The invention also relates to a method of operating an electronic device according to the appended independent method claim.

The above and other aspects, features, and advantages of the disclosure will be more apparent from the following detailed description, when taken in conjunction with the accompanying drawings, in which:.

Embodiments are described in detail with reference to the accompanying drawings. The same or similar components may be designated by the same or similar reference numerals although they are illustrated in different drawings. Detailed descriptions of constructions or processes known in the art may be omitted to avoid obscuring the subject matter of the disclosure. The embodiments and the terms used therein are not intended to limit the technology disclosed herein to specific forms, and should be understood to include various modifications and/or alternatives to the corresponding embodiments. A singular expression may include a plural expression unless they are definitely different in a context.

<FIG> is a block diagram illustrating an electronic device in a network environment, according to an embodiment.

Referring to <FIG>, an electronic device <NUM> in a network environment <NUM> may communicate with an electronic device <NUM> via a first network <NUM> (e.g., a short-range wireless communication network), or an electronic device <NUM> or a server <NUM> via a second network <NUM> (e.g., a long-range wireless communication network). The electronic device <NUM> includes a processor <NUM>, memory <NUM>, an input device <NUM>, a sound output device <NUM>, a display device <NUM>, an audio module <NUM>, a sensor module <NUM>, an interface <NUM>, a haptic module <NUM>, a camera module <NUM>, a power management module <NUM>, a battery <NUM>, a communication module <NUM>, a subscriber identification module (SIM) <NUM>, or an antenna module <NUM>.

The processor <NUM> may execute, for example, software (e.g., a program <NUM>) to control at least one other component (e.g., a hardware or software component) of the electronic device <NUM> coupled with the processor <NUM>, and may perform data processing or computation. Additionally, or alternatively, the auxiliary processor <NUM> may be adapted to consume less power than the main processor <NUM>, or to be specific to a specified function.

According to an embodiment, the auxiliary processor <NUM> (e.g., an ISP or a CP) may be implemented as part of another component (e.g., the camera module <NUM> or the communication module <NUM>) functionally related to the auxiliary processor <NUM>.

The memory <NUM> may store data used by at least one component (e.g., the processor <NUM> or the sensor module <NUM>) of the electronic device <NUM>. The data may include, for example, software (e.g., the program <NUM>) and input data or output data for a command related thereto. The memory <NUM> includes the volatile memory <NUM> or the non-volatile memory <NUM>.

The program <NUM> may be stored in the memory <NUM> as software, and includes an operating system (OS) <NUM>, middleware <NUM>, or an application <NUM>.

According to an embodiment, the camera module <NUM> may include one or more lenses, image sensors, ISPs, or flashes.

The communication module <NUM> may include one or more communication processors that are operable independently from the processor <NUM> (e.g., the AP) and supports a direct (e.g., wired) communication or a wireless communication.

The antenna module <NUM> may include a plurality of antennas. Another component (e.g., a radio frequency integrated circuit (RFIC)) other than the radiating element may be additionally formed as part of the antenna module <NUM>.

According to an embodiment, when running a first application, the processor <NUM> may control the camera module <NUM> so as to obtain at least one image, and may control the display device <NUM> so as to display at least one image on a preview screen. The first application may be an application that provides a photography function using a camera. For example, the first application may be a camera application that provides at least one of a photography function or a video recording function. However, embodiments described below may not be limited thereto, and may be applied to various applications using a camera.

The processor <NUM> determines whether a document is included in at least one image obtained by the camera module <NUM>. The processor <NUM> may determine whether a document is included in an image based on whether text exists in at least one image, and/or whether a code provided in a predetermined form (e.g., a bar code or a QR code) is included in at least one image. For example, if text, such as characters and/or numbers, or a code provided in the predetermined form, is not included in the image, the processor <NUM> may determine that a document is not included in an image. As another example, if text, such as characters and/or numbers, exists in an image, the processor <NUM> may determine whether a document is included in the image based on the shape of an object including the text in the image. For example, the processor <NUM> may determine whether a document is included in the image based on the number of segments (or sides) or the number of vertices of an object including text in the image. If the number of segments and/or the number of vertices detected from around the text do not satisfy a predetermined condition, the processor <NUM> may determine that a document is not included in the image. If the number of segments and/or the number of vertices detected from around the text satisfy a predetermined condition, the processor <NUM> may determine that a document is included in the image. When four vertices are detected from around the text, and text exists inside a quadrangular shape that connects the four vertices, the processor <NUM> may determine that a document is included in the image. As another example, if a code in a predetermined form (e.g., a bar code or a QR code) is included in the image, the processor <NUM> may determine that a document is included in the image. The above-described methods that determine whether text is included in the image are merely examples, and the embodiments of the disclosure are not limited thereto.

According to an embodiment, the processor <NUM> provides a user interface based on whether a document is included in an image. If a document is not included in at least one image, the processor <NUM> provides a first user interface for photographing an image. The first user interface includes at least one first graphic object related to photographing an image. A graphic object may include, for example, at least one of a button, an icon, a symbol, an item, or a menu, which is visually displayed. For example, the first user interface may include at least one of a first button for providing a photography function or a second button for providing a video recording function. If a document is included in at least one image, the processor <NUM> provides a second user interface for providing a document-related function. The second user interface includes at least one first graphic object related to photographing an image, and additionally includes a second graphic object indicating that a document exists in an image and at least one third graphic object that provides a document-related function. The second graphic object indicating that a document exists in an image includes at least one of an icon, a symbol, an item, or a graphic effect, which visually reports that the existence of a document is recognized. The second graphic object indicating that a document exists in an image includes a graphic effect that indicates an area where the existence of a document is recognized (or identified). The at least one third graphic object providing a document-related function includes a scan button for providing a function of scanning a document, and may also include at least one of a payment button for providing a payment function based on information included in a document, a web access button for providing a function of accessing a web page based on information included in a document, a schedule add button for providing a function of adding a schedule based on information included in a document, a contact information add button for providing a function of adding contact information based on information included in a document, or an additional information search button for providing a function of searching for additional information based on information included in a document. The above-mentioned third graphic objects are merely examples, and embodiments are not limited thereto. The second user interface may further include a first button for providing a photography function or a second button for providing a video recording function, and may additionally include a symbol indicating the existence of a document in an image, or at least one of a web access button, a payment button, or a schedule add button. Another third graphic object to be included in the second user interface may be determined based on the document type of a document included in an image, and information included in the document.

The processor <NUM> performs a photography-related function or performs a document-related function based on a user input in a state in which the second user interface is displayed. In the state in which the second user interface is displayed, if a user input to at least one first graphic object is detected, the processor <NUM> performs a photography function or a video recording function. An image obtained via the photographic function or the video recording function includes an image displayed on a preview screen. In the state in which the second user interface is displayed, if a user input to at least one third graphic object is detected, the processor <NUM> performs a document-related function. The third graphic object on which a user input is detected is an object for providing a scan function and the processor <NUM> extracts and may store a document from an image. As another example, if the third graphic object on which a user input is detected is an object for providing a web page access function, the processor <NUM> may obtain web address information from a document included in an image, and may access a corresponding web page using the obtained web address. As another example, if the third graphic object on which a user input is detected is an object that provides a payment function, the processor <NUM> may obtain payment information from a document included in an image, and may perform a payment function using the obtained payment information. As another example, if the third graphic object on which a user input is detected provides a schedule add function, the processor <NUM> may obtain schedule information from a document included in an image, and may add the obtained schedule information to a schedule management application.

According to an embodiment, the processor <NUM> determines whether a document included in an image is recognizable, and provides a user interface based on the recognizability. According to an embodiment, the processor <NUM> may determine whether the document included in the image is recognizable, based on at least one of the size of the document or the distortion level of the document. For example, if the size of the document is greater than a predetermined maximum size, or less than a predetermined minimum size, the processor <NUM> may determine that the document included in the image is not recognizable. As another example, if the distortion level of the document is beyond a predetermined level range, the processor <NUM> may determine that the document included in the image is not recognizable. If the document included in the image is recognizable, the processor <NUM> provides a second user interface. If the document included in the image is not recognizable, the processor <NUM> may provide a third user interface including an error report message, which indicates that the document is not recognizable. The error report message may include, for example, a message that requests adjustment of at least one of a photography location, a photographing direction, or a subject distance for photography (e.g., the distance between the document and the electronic device). The third user interface may include at least one first graphic object related to photographing and an error report message, and may additionally include at least one of a second graphic object indicating that a document exists in an image or at least one third graphic object that provides a document-related function. The at least one third graphic object included in the third user interface may be in a deactivated state. For example, the at least one third graphic object included in the third user interface may be deactivated so as to indicate that a document-related function is not allowed to perform.

According to an embodiment, the processor <NUM> may determine at least one document-related function based on at least one of the document type of the document included in the image or information included in the document. The processor <NUM> may determine at least one third graphic object to be included in the second user interface based on the at least one determined function. The processor <NUM> may identify the document type of the document included in the image, and may determine whether the identified type of document is a type providing a single function or a type providing a plurality of functions based on a table listing a related function for each type of document. The processor <NUM> may determine the type of document based on information included in the document or the form of the document.

For example, if a table is detected from the document included in the image, or if schedule information including time and/or place information is detected, the processor <NUM> may determine the type of document as a first type (e.g., a poster, a memo, or a calendar). As another example, if a code in a predetermined form is detected, the processor <NUM> may determine the type of document as a second type (e.g., a barcode or a QR code). As another example, if at least one of publisher information or author information is detected from the document included in the image, the processor <NUM> may determine the type of document as a third type (e.g., a book). As another example, if at least one piece of information among a user name, a company name, a company address, a phone number, a fax number, or an email address is detected from the document included in the image, the processor <NUM> may determine the type of document as a fourth type (e.g., a business card). As another example, if information corresponding to the first type to the fourth type is not detected, the processor <NUM> may determine the type of document as a fifth type (e.g., miscellaneous documents). The method of determining the type of document is merely an example, and embodiments are not limited thereto.

The table listing a related function for each type of document may be, for example, a table listing a related function for each type, such as "a document type: a first type (e.g., a poster, a memo, or a calendar), a related function: a scan function or a schedule add function", "a document type: a second type (e.g., a QR code or a barcode), a related function: a payment function or a web access function", a document type: a third type (e.g., a book), a related function: a scan function or an additional information search function", "a document type: a fourth type (e.g., a business card), a related function: a scan function or, a contact information add function", and "a document type: a fifth type (e.g., miscellaneous documents), a related function: a scan function". Embodiments of the present disclosure are not limited to the above-described examples.

If a single related function is determined for the identified type of document, the processor <NUM> may determine the type of document as a type that provides a single function, and may determine a third graphic object that is capable of performing the single function. If a plurality of related functions are determined for the identified type of document, the processor <NUM> may determine the type of document as a type that provides a plurality of functions, and may determine a third graphic object capable of performing a single function or a plurality of third graphic objects corresponding to the plurality of functions on the basis of information included in the document.

For example, if the document included in the image is the first type, the processor <NUM> may recognize text included in the document, and may determine whether schedule information (e.g., time information or place information) is included in the document. If schedule information is included in the document, the processor <NUM> may determine a scan function and a schedule add function as document-related functions, and may determine a scan button for providing a function of scanning a document and a schedule add button for providing a function of adding schedule information to a predetermined application (e.g., a calendar application or a schedule application) as third graphic objects. If schedule information is not included in the document, the processor <NUM> may determine a scan function as a document-related function, and may determine a scan button for providing a function of scanning a document as a third graphic object.

As another example, if the document included in the image is the second type (e.g., a QR code or a barcode), the processor <NUM> may analyze a code in the document, and may determine whether payment information and web address information are included. If the code in the document includes only web address information, the processor <NUM> may, in some examples that are not independently claimed, determine a web access button for accessing a corresponding web page and providing information, as a third graphic object. If the code in the document includes web address information and payment information, the processor <NUM> may determine a web access button and a payment button as third graphic objects.

As another example, if the document included in the image is the third type (e.g., a book, a research paper, or a leaflet), the processor <NUM> may recognize text included in the document, and may determine whether searchable information is included in the document. If searchable information is included in the document, the processor <NUM> may determine a scan function and an additional information search function as document-related functions, and may determine a scan button for providing a scan function and a search button for providing an additional information search function as third graphic objects.

As another example, if the document included in the image is the fourth type (e.g., a business card), the processor <NUM> may determine a scan function and a contact information add function as document-related functions, and may determine a scan button for providing a scan function and a contact information add button for providing a function of registering contact information included in a business card with an electronic device, as third graphic obj ects.

As another example, if the document included in the image is the fifth type (e.g., miscellaneous documents), the processor <NUM> may determine a scan function as a document-related function, and may determine a scan button that provides a scan function as a third graphic object. The processor <NUM> may provide the second user interface including the at least one determined third graphic object.

In the above description, the processor <NUM> determines whether a document is included in an image, and may further determine at least one of the document type of a document included in an image, or a document-related function, based on a document-related database (DB) included in the memory <NUM> of the electronic device <NUM>. The document-related DB may include various types of information (e.g., a reference value, a condition, a rule for determining a document, or a function), which are needed to determine at least one of whether a document is included in an image, the document type of a document included in an image, and a document-related function. If the processor <NUM> fails to determine at least one of whether a document is included in an image, the document type of a document included in an image, and a document-related function, using the document-related DB included in the memory <NUM>, the processor <NUM> may transmit the image to an external device (e.g., the electronic device <NUM>, the electronic device <NUM>, or the server <NUM>), and may receive at least one piece of information among whether a document is included in the image, the document type of a document included in the image, or a document-related function from the external device (e.g., the electronic device <NUM>, the electronic device <NUM>, or the server <NUM>). The document-related DB included in the memory <NUM> may be updated via interoperation with the external device (e.g., the electronic device <NUM>, the electronic device <NUM>, or the server <NUM>), and may be updated by learning.

The electronic device includes a camera, a display, and a processor. The processor is configured to determine whether a document is included in an image provided from the camera. When a document is not included in the image, the processor is configured to provide a first user interface including at least one first graphic object for photographing the image. When a document is included in the image, the processor is configured to provide a second user interface for providing a function related to the document. The second user interface includes the at least one first graphic object, a second graphic object indicating that a document exists in the image, and at least one third graphic object for performing a function related to a document.

The second graphic object includes a graphic effect indicating an area where a document is recognized in the image and may further include a predetermined symbol indicating that a document exists in the image.

When a document is included in the image, the processor may further be configured to determine whether the document is recognizable, based on whether the document satisfies a predetermined condition, and when the document is recognizable, the processor is configured to determine at least one function related to the document based on a type of document or information included in the document.

The at least one function related to the document includes a function of scanning the document, and may also include at least one of a function of searching for additional information related to the document, a payment function based on information included in the document, a web access function based on information included in the document, a schedule add function based on information included in the document, and a function of adding contact information based on information included in the document.

The second user interface includes a third graphic object that provides a function of scanning the document. When a user input to the third graphic object, which provides the function of scanning the document, is detected in the state in which the second user interface is provided, the processor is configured to extract and store only the document from the image. When a user input to the first graphic object is detected in the state in which the second user interface is provided, the processor is configured to photograph the image including the document and store the image.

The processor may determine whether at least one of a size of the document or a distortion level of the document satisfies the predetermined condition.

When the document is not recognizable, the processor may provide a third user interface including an error report message related to at least one of the size of the document or the distortion level of the document, and the third user interface may further include at least one of the at least one first graphic object or the second graphic object indicating that a document exists in the image.

The third user interface may further include at least one third graphic object for performing a function related to the document, and the at least one third graphic object may be in a deactivated state.

<FIG> is a flowchart <NUM> illustrating a process of providing various user interfaces based on whether a document is included in an image, by an electronic device, according to an embodiment. In the embodiment provided hereinafter, operations may be performed sequentially, but it is not necessary. For example, the order of operations may be changed, and at least two operations may be performed in parallel. Here, an electronic device may be the electronic device <NUM> of <FIG>. Hereinafter, at least some steps of <FIG> will be described with reference to <FIG>, <FIG>, and <FIG>.

<FIG> is a diagram illustrating an example of a user interface provided based on whether a document is included in an image, by an electronic device, according to an embodiment. <FIG> is a diagram illustrating an example of a user interface provided based on whether a document is included in an image, by an electronic device, according to an embodiment. <FIG> is a diagram illustrating an example of a user interface including at least one graphic object related to a document, in an electronic device, according to an embodiment.

Referring to flowchart <NUM> of <FIG>, an electronic device (e.g., the processor <NUM> of <FIG>) determines whether a document is included in an image provided from a camera <NUM>, in step <NUM>. The processor <NUM> may determine whether a document is included in at least one image provided from the camera <NUM> based on whether text exists in at least one image and/or a code in a predetermined form (e.g., a barcode or a QR code) is included. For example, if text or a code provided in the predetermined form is not included in an image displayed on a preview screen, the processor <NUM> determines that a document is not included in the image. As another example, if text exists in the image, the processor <NUM> detects a segment or a vertex of an object including the text in the image, and if the number of the detected segments and/or the number of vertices does not satisfy a predetermined condition, the processor <NUM> determines that a document is not included in the image. If the number of segments and/or the number of vertices detected from around the text satisfies a predetermined condition, the processor <NUM> may determine that a document is included in the image. As another example, if a code in a predetermined form is included in the image, the processor <NUM> determines that a document is included in the image. The above-described methods of determining whether text is included in an image are merely examples, and embodiments of the disclosure are not limited thereto.

If a document is not included in the image, the electronic device (e.g., the processor <NUM>) provides a first user interface for photographing an image, in step <NUM>. The first user interface includes at least one first graphic object related to photographing an image. A graphic object may include, for example, at least one of a button, an icon, a symbol, an item, or a menu, which is visually displayed. For example, as illustrated in <FIG>, if a document is not included in the image displayed on a preview screen, the processor <NUM> provides a first user interface including a photographing button <NUM> for providing a photography function and a video recording button <NUM> for providing a video recording function.

If a document is included in the image, the electronic device (e.g., the processor <NUM>) provides a second user interface for providing a document-related function, in step <NUM>. The second user interface includes at least one first graphic object related to photographing an image, a second graphic object indicating that a document exists in an image and at least one third graphic object that provides a document-related function. The second graphic object indicating that a document exists in an image may include at least one of an icon, a symbol, an item, or a graphic effect which visually reports that the existence of the document is recognized. The second graphic object indicating that a document exists in an image includes a graphic effect that indicates an area where a document is recognized and may further include a predetermined symbol. The at least one third graphic object that provides a document-related function may include, for example, at least one of a scan button for providing a function of scanning a document, a payment button for providing a payment function based on information included in a document, a web access button for providing a function of accessing a web page based on information included in a document, or an add button for providing a function of adding a schedule based on information included in a document. The above-mentioned third graphic objects are merely examples, and embodiments of the present disclosure are not limited thereto.

For example, as illustrated in <FIG>, if a document is included in the image displayed on a preview screen, the processor <NUM> provides a second user interface including a photographing button <NUM> that provides a photography function, a video recording button <NUM> for providing a video recording function, a scan button <NUM> for providing a function of scanning a document, a symbol <NUM> indicating that a document exists in an image, and a graphic effect <NUM> indicating an area where a document is recognized. The display location and/or the size of a second graphic object and/or a third graphic object to be included in the second user interface may be set and changed by a designer and/or a user.

For example, as illustrated in <FIG>, a scan button <NUM>, <NUM>, <NUM>, and <NUM>, which is a third graphic object and a symbol <NUM>, <NUM>, <NUM>, and <NUM>, which is a second graphic object and indicates the existence of a document may be displayed in an area predetermined by a designer and/or a user. The symbol that is a second graphic object and indicates the existence of a document may not trigger a predetermined function, and may be displayed on a photographing button as shown in diagram <NUM> of <FIG>. As another example, the processor <NUM> may dynamically change the display location of a second graphic object and/or a third graphic object based on the location of a document included in an image. For example, every time that the location of a document in an image is changed, the processor <NUM> may dynamically change the display location of a third graphic object based on the changed location of the document. A third graphic object included in the second user interface may be determined based on the type of a document included in an image, and/or information included in the document.

<FIG> is a flowchart <NUM> illustrating a process of performing a function corresponding to a user input when a document exists in an image, by an electronic device, according to an embodiment. Operations of <FIG>, which are described below, may be operations subsequent to step <NUM> of <FIG>. In the embodiment provided hereinafter, operations may be performed sequentially, but it is not necessary. For example, the order of operations may be changed, and at least two steps may be performed in parallel. Here, an electronic device may be the electronic device <NUM> of <FIG>. Hereinafter, at least some operations of <FIG> will be described with reference to <FIG> and <FIG>.

<FIG> is a diagram illustrating an example of performing a function corresponding to a user input, by an electronic device, according to an embodiment. <FIG> is a diagram illustrating an example of performing a function corresponding to a user input, by an electronic device, according to an embodiment.

Referring to <FIG>, an electronic device (e.g., the processor <NUM> of <FIG>) detects a user input to a second user interface, in step <NUM>. The second user interface may be the same as the second user interface described with reference to step <NUM> of <FIG>. A user input may include at least one of a single touch, a double touch, a long touch, or a pressure touch.

The electronic device (e.g., the processor <NUM>) determines whether a user input is an input with respect to a third graphic object, in step <NUM>. For example, the processor <NUM> may determine whether a user input to a third graphic object is detected, or whether a user input to a first graphic object is detected, among the graphic objects included in the second user interface.

If the user input is an input to the third graphic object, the electronic device (e.g., the processor <NUM>) performs a document-related function mapped to the third graphic object, in step <NUM>. As illustrated in <FIG>, the third graphic object includes a scan button, and may in some examples also include at least one of a payment button, a web access button, or a schedule add button. For example, as illustrated in <FIG>, if a user input <NUM> to a scan button is detected, the processor <NUM> extracts a document from an image, and performs a function of storing the extracted document <NUM>. As another example, if a user input to a payment button is detected, the processor <NUM> may obtain payment information from a document included in an image, and may perform a payment function based on the payment information. If a user input to a web access button is detected, the processor <NUM> may obtain web address information from a document included in an image, and may perform a function of accessing a corresponding web page based on the web address information. If a user input to a schedule add button is detected, the processor <NUM> may obtain schedule information from a document included in an image, and may add the obtained schedule information to an application that manages a user's schedules (e.g., a calendar application or a schedule management application).

If a user input is not an input to the third graphic object (if the user input is an input to the first graphic object), the electronic device (e.g., the processor <NUM>) performs a photography-related function mapped to the first graphic object where the user input is detected, in step <NUM>. The first graphic object may include, for example, at least one of a photographing button or a video recording button, as illustrated in <FIG>. For example, as illustrated in <FIG>, if a user input <NUM> to a photographing button is detected, the processor <NUM> performs a function of obtaining and storing an image which is provided from a camera <NUM>, and is displayed on a preview screen, as an image obtained by photography. This may be the same as a normal photography function.

<FIG> is a flowchart <NUM> illustrating a process of providing various user interfaces based on whether a document is recognizable, by an electronic device, according to an embodiment. Operations of <FIG> described below may be the detailed operations of at least some of steps <NUM>, <NUM>, and <NUM> of <FIG>. In the embodiment provided hereinafter, operations may be performed sequentially, but it is not necessary. For example, the order of operations may be changed, and at least two steps may be performed in parallel. Here, an electronic device may be the electronic device <NUM> of <FIG>. Hereinafter, at least some operations of <FIG> will be described with reference to <FIG>.

<FIG> is a diagram illustrating an example of a user interface that provides a document-related function based on the type of document, by an electronic device, according to an embodiment. <FIG> is a diagram illustrating an example of a user interface that provides a document-related function based on the type of document, by an electronic device, according to an embodiment. <FIG> is a diagram illustrating an example of a user interface that provides a document-related function based on the type of document, by an electronic device, according to an embodiment that is not independently claimed. <FIG> is a diagram illustrating an example of a user interface that provides a document-related function based on the type of document, by an electronic device, according to an embodiment that is not independently claimed. <FIG> is a diagram illustrating an example of a user interface that provides a document related function based on the type of document, by an electronic device, according to an embodiment.

Referring to flowchart <NUM> of <FIG>, an electronic device (e.g., the processor <NUM> of <FIG>) operates a camera (e.g., the camera module <NUM> of <FIG>), and obtains at least one image, in step <NUM>. When running a first application, the processor <NUM> operates the camera <NUM>, and may obtain at least one image from the camera. The processor <NUM> may control the display device <NUM> so as to display the at least one image obtained from the camera module <NUM> on a preview screen. The first application may be an application that provides a photography function using a camera. For example, the first application may be a camera application that provides at least one of a photography function or a video recording function.

The electronic device (e.g., the processor <NUM>) determines whether a document is included in an image, in step <NUM>. The step of determining whether a document is included in the image may be at least partially the same as step <NUM> of <FIG>. The processor <NUM> may determine whether a document is included in the image based on a document-related DB included in a memory <NUM> of the electronic device <NUM>. The document-related DB may include various types of information (e.g., a reference value, a condition, a rule of determining a document, or a function) needed to determine whether a document is included in the image. If it is identified that determining whether a document is included in the image is difficult using the document-related DB included in the memory <NUM>, the processor <NUM> may transmit the image to an external device (e.g., the electronic device <NUM>, the electronic device <NUM>, or the server <NUM>) and may request the external device to determine whether a document is included in the image. In response to the request, the processor <NUM> may receive a response signal including information indicating whether a document is included in the image, from the external device (e.g., the electronic device <NUM>, the electronic device <NUM>, or the server <NUM>). The response signal may additionally include at least one information associated with the document type of a document included in the image or a document-related function, in addition to the information indicating whether a document is included in the image.

If it is determined that a document is not included in the image, the electronic device (e.g., the processor <NUM>) performs step <NUM> of <FIG>.

If it is determined that a document is included in the image, the electronic device (e.g., the processor <NUM>) determines whether the document included in the image is recognizable, in step <NUM>. According to an embodiment, the processor <NUM> may determine whether the document included in the image is recognizable, based on at least one of the size of the document or the distortion level of the document. For example, if the size of the document is greater than a predetermined maximum size, or less than a predetermined minimum size, the processor <NUM> may determine that the document included in the image is not recognizable. As another example, if the distortion level of the document is beyond a predetermined level range, the processor <NUM> may determine that the document included in the image is not recognizable.

If the document included in the image is recognizable, the electronic device (e.g., the processor <NUM>) determines whether the document type of the document is a type providing a plurality of functions in step <NUM>. The processor <NUM> may identify the document type of the document included in the image, and may determine whether a single document-related function or a plurality of document-related functions are used based on the type of document. For example, based on the information included in the document or the form of the document, the processor <NUM> may determine the type of document, and may determine whether a single document-related function or a plurality of document-related functions are used based on a table listing a related function for each type of document. The table listing a related function for each type of document may be, for example, a table listing a related function for each type, such as "a document type: a first type, a related function: a scan function or a schedule add function", "a document type: a second type, a related function: a payment function or a web access function", "a document type: a third type, a related function: a scan function or an additional information search function", "a document type: a fourth type, a related function: a scan function, a contact information add function", and "a document type: a fifth type, a related function: a scan function". If the identified type of document is any one of the first type to the fourth type, the processor <NUM> may determine the type of document as a type providing a plurality of functions. If the identified type of document is the fifth type, the processor <NUM> may determine the type of document as a type that is incapable of providing a plurality of functions (a type providing a single function).

If the type of document is a type providing a plurality of functions, the electronic device (e.g., the processor <NUM>) determines a single or a plurality of functions as document-related functions based on information included in the document, in step <NUM> and provides a second user interface including at least one third graphic object for performing the document-related function, in step <NUM>. For example, if the document included in the image is the first type, the processor <NUM> recognizes text included in the document, and may determine whether schedule information (e.g., time information or place information) is included in the document.

If schedule information is not included in the document, as illustrated in <FIG>, the processor <NUM> determines a scan function as a document-related function, and provides a second user interface including a scan button <NUM> and a graphic effect <NUM> indicating an area where the document is recognized. If schedule information <NUM> is included in the document, as illustrated in <FIG>, the processor <NUM> determines a scan function and a schedule add function as document-related functions, and may provide a second user interface including a scan button <NUM>, a schedule add button <NUM>, and a graphic effect <NUM> indicating an area where the document is recognized.

As another example that is not independently claimed, if the document included in the image corresponds to the second type (e.g., a QR code or a barcode), the processor <NUM> may analyze a code in the document, and may determine whether payment information and web address information are included. If the code in the document includes only web address information, as illustrated in <FIG>, the processor <NUM> determines a web access function as a document-related function, and provides a second user interface including a web access button <NUM>. If the code in the document includes web address information and payment information, as illustrated in <FIG>, the processor <NUM> determines a web access function and a payment function as document-related functions, and provides a second user interface including a web access button <NUM> and a payment button <NUM>.

As another example, if the document included in the image corresponds to the third type (e.g., a book, a research paper, or a leaflet), the processor <NUM> may recognize text included in the document, and may determine whether searchable information is included in the document. If the searchable information is included in the document, as illustrated in <FIG>, the processor <NUM> determines a scan function and an additional information search function as document-related functions, and may provide a second user interface including a scan button <NUM> and an additional information search button <NUM>. If the searchable information is not included in the document, the processor <NUM> may determine a scan function as a document-related function, and may provide a second user interface including a scan button.

As another example, if the document included in the image corresponds to the fourth type (e.g., a business card), the processor <NUM> may determine a scan function and contact information add button as document-related functions, and may provide a second user interface including a scan button and a contact information add button. The above-described second user interfaces may further include at least one first graphic object related to photographing an image and a second graphic object indicating the existence of a document in an image.

Referring back to <FIG>, if the type of document is a type providing a single function, the electronic device (e.g., the processor <NUM>) determines a single function based on the information included in the document as a document-related function, in step <NUM>, and provides a second user interface including at least one third graphic object for performing the document-related function determined, in step <NUM>. For example, if the document included in the image corresponds to the fifth type (e.g., miscellaneous documents), the processor <NUM> may determine a scan function as a document-related function, and may provide a second user interface including a scan button.

If the document included in the image is not recognizable, the electronic device (e.g., the processor <NUM>) provides a third user interface including an error report message indicating that the document is not recognizable, in step <NUM>. The error report message may include, for example, a message that requests adjustment of at least one of a photography location, a photographing direction, or a subject distance for photography (e.g., the distance between a document and an electronic device). The error report message may include, for example, at least one of information associated with the size of the document, information related to the distortion of the document, information associated with a request for adjusting the photography location, information associated with a request for adjusting the subject distance for photography, or information associated with a request for adjusting the photographing direction. The third user interface may include at least one first graphic object related to photographing and an error report message, and may additionally include at least one of a second graphic object indicating that a document exists in an image and at least one third graphic object that provides a document-related function. The at least one third graphic object included in the third user interface may be in a deactivated state. For example, the at least one third graphic object included in the third user interface may be deactivated so as to indicate that the document-related function is not allowed to perform.

<FIG> is a flowchart <NUM> illustrating a process of providing various user interfaces based on a predetermined condition, by an electronic device, according to an embodiment. Operations of <FIG> described below may be the detailed operations of at least some of steps <NUM> to <NUM> of <FIG>. In the embodiment provided hereinafter, steps may be performed sequentially, but it is not necessary. For example, the order of steps may be changed, and at least two steps may be performed in parallel. Here, an electronic device may be the electronic device <NUM> of <FIG>. Hereinafter, at least some operations of <FIG> will be described with reference to <FIG>, and <FIG>.

<FIG> is a diagram illustrating an example of a user interface including an error report message related to the size of a document, by an electronic device, according to an embodiment. <FIG> is a diagram illustrating an example of a user interface including an error report message related to the size of a document, by an electronic device, according to an embodiment. <FIG> is a diagram illustrating an example of a user interface including an error report message related to the size of a document, by an electronic device, according to an embodiment. <FIG> is a diagram illustrating an example of a user interface provided based on the distortion level of a document by an electronic device, according to an embodiment. <FIG> is a diagram illustrating an example of a user interface provided based on the distortion level of a document by an electronic device, according to an embodiment. <FIG> is a diagram illustrating an example of a user interface provided based on the distortion level of a document by an electronic device, according to an embodiment.

Referring to flowchart <NUM> of <FIG>, an electronic device (e.g., the processor <NUM> of <FIG>) determines whether the size of a document is less than or equal to a predetermined maximum size, in step <NUM>. For example, the processor <NUM> may determine whether the entirety of a document is included in an image or a part of the document is included in the image, based on the number of segments or vertices of the document, and determines whether the size of the document is less than or equal to the predetermined maximum size. If all of the four segments or the four vertices of the document are detected, the processor <NUM> determines that the entirety of the document is included in the image, and determines that the size of the document is less than or equal to the predetermined maximum size. If the four segments or four vertices are not detected (e.g., three or fewer segments or three or fewer vertices are detected), the processor <NUM> determines that a part of the document is included in the image, and determines that the size of the document is greater than the predetermined maximum size.

If the size of the document is greater than the predetermined maximum size, the electronic device (e.g., the processor <NUM>) provides a third user interface including an error report message that requests decreasing the size of the document, in step <NUM>. The error report message that requests decreasing the size of the document may be a message of requesting adjustment of the photography location and/or the subject distance for photography of a camera so that the entirety of the document is included in the image since the size of the document is greater than the predetermined maximum size.

For example, as illustrated in <FIG>, if only a part of a calendar is included in the image, the processor <NUM> provides a third user interface including an error report message <NUM> which requests adjustment to enable the entirety of the calendar to be included in the image and to store the calendar without distortion. According to an embodiment, the third user interface may include at least some of graphic elements which are included in the third user interface, which have been described in step <NUM> of <FIG>.

If the size of the document is less than or equal to the predetermined maximum size, the electronic device (e.g., the processor <NUM>) determines whether the size of the document is greater than or equal to a predetermined minimum size, in step <NUM>. According to an embodiment, if the entirety of the document is included in the image, the processor <NUM> may measure the size of the entire document, and may compare the measured size of the entire document and the predetermined minimum size.

If the size of the document is less than the predetermined maximum size, the electronic device (e.g., the processor <NUM>) provides a third user interface including an error report message that requests increasing the size of the document, in step <NUM>. The error report message that requests increasing the size of the document may be a message that requests adjustment of the distance to the document for photography to enable the size of the document to be increased since the size of the document is less than the predetermined minimum size.

For example, as illustrated in <FIG>, if a small memo, which is smaller than the predetermined minimum size, is included in the image, the processor <NUM> provides a third user interface including an error report message <NUM>, which requests adjusting the distance to the memo for photography to be closer so that the memory is stored without distortion. As another example, as illustrated in <FIG>, if a small memo, which is smaller than the predetermined minimum size, is included in the image, the processor <NUM> provides a third user interface including an error report message <NUM> and a graphic element <NUM>. The error report message <NUM> requests adjusting the distance to the memo for photography to be closer so that the memory is stored without distortion, and the graphic element <NUM> indicates a recommended size and/or area associated with the memo. The third user interface may include at least some of graphic elements which are included in the third user interface, which has been described in step <NUM> of <FIG>.

If the size of the document is less than or equal to the predetermined minimum size, the electronic device (e.g., the processor <NUM>) determines whether a distortion level is greater than a first predetermined level, in step <NUM>. The distortion level may be, for example, a degree of distortion of the document included in the image. According to an embodiment, the distortion level may be determined based on the angle of at least one of the four vertices of the document. The distortion level may be determined based on the four segments of the document, or the angle of at least two of the plurality of segments of the document. The first predetermined level may be a level at which a document is scanned and stored without performing an additional process related to distortion (e.g., compensation for distortion), or a level in which text included in a document is recognizable without compensation for distortion. The first predetermined level may be set and/or changed by a designer and/or a user. For example, the first predetermined level may be defined to include the case in which the angles of the four vertices of the document are all greater than or equal to a first threshold angle (e.g., <NUM> degrees). For example, if the angles of the four vertices of the document are all greater than or equal to the first threshold angle, the processor <NUM> may determine that the distortion level of the document is less than or equal to the first predetermined level. If the angle of at least one of the four vertices of the document is less than the first threshold angle, the processor <NUM> may determine that the distortion level of the document is greater than the first predetermined level. This is merely an example provided for ease of understanding, and embodiments are not limited thereto. For example, embodiments may determine the distortion level of the document using various methods known to those skilled in the art.

If the distortion level of the document is less than or equal to the first predetermined level, the electronic device (e.g., the processor <NUM>) proceeds with step <NUM> of <FIG> and subsequent operations. For example, if the distortion level of the document is less than or equal to the first predetermined level, the processor <NUM> may perform at least one of steps <NUM>, <NUM>, <NUM>, or <NUM> of <FIG>, may determine a document-related function based on the type of document, and may provide a second user interface including at least one third graphic object for performing the document-related function. For example, as illustrated in <FIG>, if the distortion level of the document is less than or equal to the first predetermined level, the processor <NUM> may provide a second user interface including the scan button <NUM>.

If the distortion level of the document is greater than the first predetermined level, the electronic device (e.g., the processor <NUM>) determines whether the distortion level is less than a second predetermined level, in step <NUM>. The second predetermined level may be a level in which a document is scanned and stored after compensation for distortion of the document, or a level in which text included in the document is recognizable after compensation for distortion of the document. The second predetermined level may be set and/or changed by a designer and/or a user. For example, the second predetermined level may be set to a value greater than the first predetermined level. For example, the second predetermined level may be defined to include the case in which the angles of the four vertices of the document are all greater than a second threshold value (e.g., <NUM> degrees), and the angle of at least one of the vertices is less than the first threshold angle. For example, if the angles of the four vertices of the document are all greater than the second threshold value and the angle of at least one vertex is less than the first predetermined angle, it is determined that the distortion level of the document is less than or equal to the second predetermined level and greater than the first predetermined level. If the angle of at least one of the four vertices of the document is less than the second threshold angle, the processor <NUM> may determine that the distortion level of the document is greater than the second predetermined level.

If the distortion level of the document is less than the second predetermined level, and is greater than the first predetermined level, the electronic device (e.g., the processor <NUM>) determines a document-related function based on the type of document, in step <NUM>. The step of determining a document-related function based on the type of document may be the same as steps <NUM> and <NUM> of <FIG>, or may be the same as steps <NUM> and <NUM> of <FIG>.

The electronic device (e.g., the processor <NUM>) provides a fourth user interface including at least one third graphic object for performing the determined document-related function and a distortion report message, in step <NUM>. The fourth user interface may include at least one graphic element included in the second user interface, and may additionally include a distortion report message.

For example, as illustrated in <FIG>, if the distortion level of the document is less than the second predetermined level, and is greater than the first predetermined level, the processor <NUM> provides a fourth user interface including a scan button <NUM> and a distortion report message <NUM>. The distortion report message may include a message indicating that a photographing direction and/or the orientation of the document needs to be changed in order to clearly scan the document without distortion. According to an embodiment, if a user input to a scan button is detected in the fourth user interface, the processor <NUM> may obtain the document in the image, may compensate for the distortion of the obtained document, and may store the document of which distortion is compensated.

If the distortion level of the document is greater than or equal to the second predetermined level, the electronic device (e.g., the processor <NUM>) provides a third user interface including a distortion report message, in step <NUM>. The distortion report message may include a message indicating that a photographing direction and/or the orientation of the document needs to be changed in order to clearly scan the document without distortion. According to an embodiment, the third user interface may include at least some of graphic elements which are included in the third user interface, which has been described in step <NUM> of <FIG>. According to an embodiment, the third user interface may include a deactivated scan button.

For example, as illustrated in <FIG>, if the distortion level of the document is greater than the second predetermined level, the processor <NUM> provides a third user interface including a deactivated scan button <NUM> and a distortion report message <NUM>.

A method of an electronic device includes determining whether a document is included in an image provided from a camera. When a document is not included in the image, a first user interface is provided including at least one first graphic object for photographing the image. When a document is included in the image, a second user interface is provided for providing a function related to the document. The second user interface includes the at least one first graphic object, a second graphic object indicating that a document exists in the image, and at least one third graphic object for performing the function related to a document.

Providing the second user interface that provides the function related to the document may include, when a document is included in the image, determining whether the document is recognizable, based on whether the document satisfies a predetermined condition. When the document is recognizable, at least one function related to the document is determined based on a type of document or information included in the document.

The at least one function related to the document includes a function of scanning the document, and may in some cases also include at least one of a function of searching for additional information related to the document, a payment function based on information included in the document, a web access function based on information included in the document, a schedule add function based on information included in the document, and a function of adding contact information based on information included in the document.

The second user interface includes a third graphic object that provides a function of scanning the document, and the method further includes, when a user input to the third graphic object, which provides the function of scanning the document, is detected in the state in which the second user interface is provided, extracting and storing only the document from the image. When a user input to the first graphic object is detected in the state in which the second user interface is provided, the image including the document is photographed and stored.

Determining whether the document is recognizable may include determining whether at least one of a size of the document or a distortion level of the document satisfies the predetermined condition.

The method may further include providing, by the processor, a third user interface including an error report message related to at least one of a size of the document or a distortion level of the document, when the document is not recognizable. The third user interface may further include at least one of the at least one first graphic object and the second graphic object indicating that a document exists in the image.

The third user interface may further include at least one third graphic object for performing a function related to the document, and the at least one third graphic object is in a deactivated state.

It should be appreciated that various embodiments of the present disclosure and the terms used therein are not intended to limit the technological features set forth herein to particular embodiments and include various changes or replacements for a corresponding embodiment.

For example, according to an embodiment, a module may be implemented in a form of an application-specific integrated circuit (ASIC).

According to an embodiment, a method may be included and provided in a computer program product.

Claim 1:
An electronic device (<NUM>), comprising:
a camera (<NUM>);
a display (<NUM>); and
a processor (<NUM>),
wherein the processor (<NUM>) is configured to:
determine whether a document is included in an image provided from the camera (<NUM>);
when a document is not included in the image, provide a first user interface to the display (<NUM>) that includes the image and at least one first graphic object for photographing the image;
when a document is included in the image, provide a second user interface to the display (<NUM>) for providing a function related to the document, wherein the second user interface comprises the image, the at least one first graphic object, a second graphic object indicating that the document exists in the image, and at least one third graphic object for performing a function related to the document, wherein the second graphic object comprises a graphic effect indicating an area where the document is recognized in the image, wherein the at least one third graphic object provides a function for scanning the document;
when a user input to the third graphic object for scanning the document is detected in a state in which the second user interface is provided, extract from the image only the document in the area where the document is recognized in the image; and
when a user input to the first graphic object for photographing the image is detected in a state in which the second user interface is provided, photograph the image including the document.