INFORMATION PROCESSING SYSTEM, PROCESSING METHOD, AND RECORDING MEDIUM

An information processing system communicably connectable to one or more apparatuses and configured to execute a plurality of processes, includes: a data storing unit configured to store, in response to a user operation on a first apparatus, setting information associating identification information identifying content of a particular process among the plurality of processes and the particular process; an area identifying unit configured to identify an area including the identification information from image data received from a second apparatus; and a processing unit configured to execute the particular process, based on the identified arca and the setting information.

TECHNICAL FIELD

The present disclosure relates to an information processing system, a processing method, and a recording medium.

BACKGROUND ART

Processing of forms such as expense claim forms, invoices, bills, or purchase order forms is common in daily work of an organization such as a company or a group. A person in charge in the organization performs the task of extracting handling dates, case numbers, amounts of money, or the like from forms or tabulating handling dates, case numbers, amounts of money, or the like in forms on a daily basis.

A technique for digitizing such a form has been devised (see, for example, PTL1). PTL1 discloses a technique for detecting a highlighted area including an area filled with a chromatic color in a read image of an original document and a circled area including an area circled by the chromatic color, and classifying the highlighted area as an area for a first process and the circled area as an area for a second process.

CITATION LIST

Patent Literature

Japanese Unexamined Patent Application Publication 2019-096178-A

SUMMARY OF INVENTION

Technical Problem

In the related art, however, it is difficult for a user to associate marks with processes. For example, if the user fails to register the correspondence between marks and processes on a predetermined sheet by marking the sheet with a marker pen or the like, the person in charge performs the task of setting the correspondence between the marks and the processes in a file or the like.

Solution to Problem

According to an aspect of the present disclosure, an information processing system communicably connectable to one or more apparatuses and configured to execute a plurality of processes includes a data storing unit that stores setting information in which identification information identifying content of a particular process among the plurality of processes and the particular process are associated with each other, in response to a user's operation on a first apparatus; an area identifying unit that identifies an area including the identification information from image data received from a second apparatus; and a processing unit that executes the particular process, based on the identified area and the setting information.

Advantageous Effects of Invention

It is possible to provide an information processing system that allows a user to associate identification information to be added to a predetermined sheet with processes.

DESCRIPTION OF EMBODIMENTS

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the present invention. As used herein, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. In describing embodiments illustrated in the drawings, specific terminology is employed for the sake of clarity. However, the disclosure of this specification is not intended to be limited to the specific terminology so selected and it is to be understood that each specific element includes all technical equivalents that have a similar function, operate in a similar manner, and achieve a similar result.

First Embodiment

The following describes, as an exemplary embodiment of the present disclosure, a data output system and a processing method performed by the data output system.

Overview of Processes or Operation

Processing of forms such as expense claim forms, invoices, bills, or purchase order forms is common in an organization such as a company or a group. A person in charge in the organization performs the task of extracting dates, case numbers, amounts of money, or the like from forms or tabulating dates, case numbers, amounts of money, or the like in forms on a daily basis. The processed forms are stored after filing, but the storage space is costly. While forms have been gradually being converted into electronic files, tasks such as renaming file names automatically assigned to the files of the forms are time-consuming to the person in charge.

Many of systems or services available for automating billing and invoicing management or order processing are expensive, and not many small or medium companies can invest in such systems or services. Various forms are used by organizations, and even items of the same form to be used for processes may be different from organization to organization. Accordingly, in the current situation, processing of such a form is inevitably performed manually.

In this embodiment, the information processing system receives, from a device, image data in which a predetermined mark (such as a marker color or a shape) is associated with a process corresponding to the mark. This enables the user to associate the mark with the process. The information processing system further receives, from the device, image data of a form on which the mark is formed. The information processing system associates the mark, the process, and area information in the form with a setting file. When the form is scanned by the device, the information processing system executes a process associated with an area of the form identified by the area information.

Accordingly, the information processing system is capable of performing various processes designated by the user on specific areas on a document.

An overview of processes or operation performed by a data output system100will be described with reference toFIGS.1,2A, and2B.FIG.1illustrates processes or operation for associating a mark with a process in the data output system100.FIGS.2A and2Billustrate processes or operation for creating a file from acquired data obtained by the data output system100from a form.

(1) The data output system100is capable of cooperating with an external service system40. The data output system100cooperates with the external service system40via an application. An application operating in the data output system100may be simply referred to as an “application” or “app”, and an application operating in the external service system40may be referred to as a cooperation application. In this case, the application ID of the application and the application ID of the cooperation application are associated with each other, and the correspondence between the application ID of the application and the application ID of the cooperation application is stored in the data output system100.

(2) An administrator 8 of the data output system100prints a registration sheet registered in an information processing system10using a device20, and colors the registration sheet with a color corresponding to a process using a marker pen. For example, the administrator 8 can associate processes with colors such that optical character recognition (OCR) is associated with green, optical mark recognition (OMR) is associated with blue, and trimming is associated with red. The device20scans a registration sheet to generate image data (first image data), and transmits the image data to the information processing system10.

(3) The information processing system10detects a correspondence between colors and processes from the image data, and registers the colors and the processes in a setting file in association with each other. The information processing system10stores the setting file in the external service system40.

Next, a description will be given with reference toFIGS.2A and2B.FIG.2Aillustrates setting of area information in a form, andFIG.2Billustrates acquisition of data from the form using the area information.

(4) The administrator 8 uses a form 7, which is usually used by users in the organization, and designates, with a marker pen, which of the processes (e.g., OCR, OMR, and trimming) is to be performed on each area of the form 7. For example, the administrator 8 may use a marker pen to fill an area of the form 7 with a color or circle an area of the form 7 in a color. The device20reads the form 7 with the marked areas to generate image data (second image data) and transmits the image data (second image data) to the information processing system10. The form 7 to be marked may include an item filled in with a character or the like, or an item only, or may be blank.

(5) The information processing system10identifies, from the image data of the form, an area colored in the same color as a color registered in a setting file acquired from the external service system40, and stores area information of the area in the setting file in association with the corresponding color and process in the setting file. The area information is, for example, the coordinates of diagonal vertices of a rectangular area in the image data. The information processing system10stores the setting file in the external service system40.

(6)-1 The administrator 8 connects a terminal apparatus30to the external service system40, downloads the setting file, and inputs the language, the date and time, and the like in the form to the setting file to complement the setting file.

(6)-2 The administrator 8 further connects the terminal apparatus30to the information processing system10and performs an operation of associating the application ID of an application with the setting file. The administrator 8 can associate application IDs of any number of applications with the setting file. As a result, an icon for accepting selection of an application and area information on which image processing is to be performed are associated with each other.

(7) To submit the form 7 to the organization for reimbursement of expenses or the like, a user 6 in the organization presses the icon of the application corresponding to the form 7 on the device20. The user 6 is allowed to easily select an application from a plurality of form names or the like. The user 6 causes the device20to read a form with items filled in by handwriting or a printed form. The device20reads the application ID of the application and the form to generate image data (third image data, such as a Portable Document Format (PDF) file), and transmits the image data to the information processing system10.

(8) The information processing system10identifies a setting file using the application ID, and acquires the setting file from the external service system40. The information processing system10acquires (for example, clips), for each process, an area of area information included in the acquired setting file from the image data. Accordingly, the area of the form registered by the administrator 8 is acquired.

(9) The information processing system10performs, for each area, a process associated with the area. As a result, a character code such as a character, the presence or absence of a check, image data, or the like is obtained for each area. In the following, these pieces of data are simply referred to as “acquired data”.

(10) The information processing system10outputs the acquired data to an electronic file (e.g., a comma-separated value (CSV) file) and transmits the file and the image data of the form to the external service system40. In response to designating the application ID of the cooperation application, the information processing system10can designate the cooperation application and transmit the file and the image data.

(11) The external service system40registers the image data and the file in a folder (or directory) designated by the administrator 8. Since the file includes information to be used for reimbursement, the person in charge can perform the reimbursement process.

As described above, the data output system100according to this embodiment enables the administrator 8 to register a mark (color) and the content of a process. The device20reads a form with marked areas, and the information processing system10acquires image data of the form. Thus, a process can be stored in association with each area of the form. The information processing system10can acquire an area from the image data of the form transmitted from the device20and perform a process corresponding to the area. Accordingly, it is possible to associate an area for image processing and a process with each of any number of applications, and the user may select an application depending on the type of the form, for example, to perform a desired process (such as OCR processing) on a desired area of image data generated by scanning.

Supplementary Information Related to Correspondence between Color and Process The series of processes described with reference toFIGS.1,2A, and2Bwill be divided into tasks1to3to provide supplementary information.FIG.3illustrates registration (task 1) of a correspondence between a color and a process using a registration sheet.

FIG.3(a)illustrates the registration sheet before coloring is applied.

FIG.3(b)illustrates the registration sheet after coloring is applied. The registration sheet illustrated inFIG.3(b)is scanned by the device20.

FIG.3(c)illustrates a portion of a setting file in which colors and processes are associated with each other. Two methods are conceivable for associating colors with processes.

In a first method, information indicating whether to associate the coordinates (in the registration sheet) at which a color is acquired with the process obtained from the coordinates is stored in the information processing system10in advance. Example parameters prepared in advance are as follows: Color extracted at coordinates (X coordinates: 100, 200; Y coordinates: 1, 50) for OCR; Color extracted at coordinates (X coordinates: 100, 200; Y coordinates: 51, 100) for masking (OMR in Table 4); and Color extracted at coordinates (X coordinates: 100, 200; Y coordinates: 101, 150) for trimming. In a second method, the administrator 8 associates colors with processes on the management site.FIG.4illustrates an example of a management site150for associating a color with a process. The management site150is provided as a web page or a web application from the information processing system10to the terminal apparatus30. The administrator 8 can register colors for processes using radio buttons151. In the illustrated example, the user uses three marker pen colors.

In either method, as illustrated inFIG.3(c), a numerical value identifying the marker color (green) and OCR are associated with each other. A numerical value identifying the marker color (blue) and OMR are associated with each other. A numerical value identifying the marker color (red) and trimming are associated with each other.

FIG.5illustrates a correspondence (task 2) between area information and a process using the original of a form.

FIG.5(a)illustrates the original of the form.

FIG.5(b)illustrates the form after coloring is applied. The form illustrated inFIG.5(b)is scanned by the device20.

FIG.5(c)illustrates a portion of a setting file in which area information and processes are associated with each other.

As illustrated inFIG.5(d), colors and processes are associated with each other in the setting file. The information processing system10detects a color of the registration sheet (registered in the setting file) from the form, and associates the coordinates of the area where the color is detected with the process in the setting file as area information.

In this embodiment, accordingly, the administrator 8 can change a color used for an area of the form to change the process to be performed on the area.

Ex1: colored in blue→OCR, colored in red→masking

Ex2: colored in blue→beginning of file name, colored in red→end of file name

FIG.6illustrates a process (task 3) to be performed on an area in a form using the form.

FIG.6(a)illustrates the original with items filled in by the user.

FIG.6(b)illustrates processes to be performed on respective areas of the form. Since OCR and file name assignment are registered as processes for a title area140of the form, the title is the file name. Since masking is registered as a process for a preliminary application area141of the form, the area corresponding to the preliminary application is masked. Since trimming is registered as a process for a remarks area142of the form, the remarks area is acquired as image data.

Terms Used Herein

A mark is a type of way how the mark is distinguished from others. In one or more embodiments, a color is an example of a mark. Other examples of the mark include a character, an alphabet, and a sign.

A first apparatus is a device or a terminal apparatus of an administrator. A second apparatus is, for example, a device.

The phrase “in response to a user's operation” includes a case where a user performs setting using a registration sheet and a case where a user performs setting using a management screen.

Setting information is, for example, information associating a color with the content of a process (see, for example, Table 4).

The administrator 8 is a system administrator or the like on the customer side of the data output system100and is a person who performs setting related to the data output system100for a general user. A user is a general user on the customer side. A general user may perform the tasks of the administrator 8, and the administrator 8 and the general user are not distinguished from each other.

A form generally refers to a ledger, a slip, or the like. A form is also referred to as a business sheet containing blank spaces to be filled in by the user. While one or more embodiments mainly describe a form, any document readable by the device20may be used. For example, the form may be replaced with a document, a paper document, a written statement, a report, a newspaper, a magazine, or the like. Some forms are not printed on paper media (some forms need not be read by the device20).

Any type of image data may be to be processed, and examples of the type of image data include PDF, Joint Photographic Experts Group (JPEG), Portable Network Graphics (PNG), Tagged Image File Format (TIFF), and bitmap (BMP).

In one or more embodiments, the term “setting file” refers to a file in which a process and area information are set, and the term “file” refers to a file to which acquired data is output. A file is a collection of data and is a basic unit of data managed by a computer.

Outputting data refers to enabling a user to visually check the data. In one or more embodiments, data is output to a file, for example.

The web application is software or a mechanism thereof that operates in coordination with a program written in a programming language (e.g., JavaScript (registered trademark)) operating on a browser and a program on the web server and that is executed on the browser.

Example System Configuration

First, the system configuration of the data output system100according to this embodiment will be described with reference toFIG.7.FIG.7illustrates an example system configuration of the data output system100according to this embodiment.

The data output system100illustrated inFIG.7includes the information processing system10and the device20. The information processing system10is communicably connected to the device20via a wide-area network N1such as the Internet. The data output system100communicates with the external service system40, which is an external system. In addition, any terminal apparatus30is capable of communicating with the information processing system10via the network N1and a network N2.

The information processing system10is implemented by one or more information processing apparatuses and provides various services implemented by a series of processes in cooperation with the external service system40, such as a cloud service, via the network N1. Specific examples of the services provided by the information processing system10according to this embodiment will be described below. The information processing system10may be implemented by cloud computing or by a single information processing apparatus. The term “cloud computing” refers to internet-based computing where resources on a network are used or accessed without identifying specific hardware resources. The information processing system10may be located on the Internet or in an on premise environment. A series of processes is provided by one application. A series of processes is also referred to as a “process flow” or a “workflow”.

The device20is any electronic device used by a user. Examples of the device20include an image processing apparatus such as a multifunction peripheral (MFP), an image forming apparatus, a personal computer (PC), a projector, an electronic whiteboard, and a digital camera. The device20is connected to the network N2. A user can use the device20to use various services provided by at least one of the information processing system10or the external service system40.

In the following, a plurality of devices20are distinguished from each other by using subscripts such as “device201” and “device202”.

The terminal apparatus30is, for example, a desktop PC, a laptop PC, a smartphone, a tablet terminal, or the like, which is used by the administrator 8 or a user. Other examples of the terminal apparatus30include an apparatus on which a web browser is operable. The terminal apparatus30is connected to the network N2. The administrator 8 or a user is able to operate the terminal apparatus30to use various services provided by the information processing system10or the external service system40, and is also able to set a cooperation application or an application.

While the device20and the terminal apparatus30will be described separately for convenience of description, the device20and the terminal apparatus30may be collectively referred to as a device.

In the following, a plurality of terminal apparatuses30are distinguished from each other by using subscripts such as “terminal apparatus301” and “terminal apparatus302”.

The external service system40includes one or more information processing apparatuses that provide services via the network NI in response to the execution of a cooperation application. In response to the execution of a cooperation application, data is managed or processed, and the data itself or processing results are provided as services.

Examples of the external service system40include a cloud service system and an application service provider (ASP), and may provide various external services via networks. In this embodiment, a storage service will be described as an example of the services. The external service system40may be located on the Internet or in an on premise environment.

In the following, when a plurality of external service systems40are distinguished from each other, an “external service system401” and an “external service system402” in the illustrated example are represented by an external service system40A and an external service system40B, respectively.

Example Hardware Configuration

The hardware configuration of the information processing system10and the terminal apparatus30included in the data output system100according to this embodiment will be described with reference toFIG.8.

Information Processing System and Terminal Apparatus

FIG.8illustrates an example hardware configuration of the information processing system10and the terminal apparatus30according to this embodiment. As illustrated inFIG.8, the information processing system10and the terminal apparatus30are constructed by a computer and include a central processing unit (CPU)501, a read-only memory (ROM)502, a random access memory (RAM)503, a hard disk (HD)504, a hard disk drive (HDD) controller505, a display506, an external device connection interface (I/F)508, a network I/F509, a bus line510, a keyboard511, a pointing device512, a digital versatile disk rewritable (DVD-RW) drive514, and a medium I/F516.

The CPU501controls the overall operation of the information processing system10and the terminal apparatus30. The ROM502stores programs such as an initial program loader (IPL) to boot the CPU501. The RAM503is used as a work area for the CPU501. The HD504stores various data such as a program. The HDD controller505controls reading or writing of various data from or to the HD504under control of the CPU501. The display506displays various information such as a cursor, a menu, a window, text, or an image. The external device connection I/F508is an interface for connecting to various external devices. Examples of the external devices include, but are not limited to, a universal serial bus (USB) memory and a printer. The network I/F509is an interface for performing data communication using the network N2. The bus line510is, for example, an address bus or a data bus for electrically connecting the elements such as the CPU501illustrated inFIG.8.

The keyboard511is an example of an input device provided with a plurality of keys for allowing a user to enter characters, numerical values, or various instructions. The pointing device512is an example of an input device that allows a user to select or execute a specific instruction, select a target for processing, or move a cursor being displayed. The DVD-RW drive514controls reading or writing of various data from or to a DVD-RW513, which is an example of a removable recording medium. In another example, a DVD-R may be used instead of the DVD-RW513. The medium I/F516controls reading or writing (storing) of data from or to a recording medium515such as a flash memory.

Device

FIG.9is a hardware configuration diagram of an image forming apparatus, which is an example of the device20. As illustrated inFIG.9, the image forming apparatus includes a controller910, a short-range communication circuit920, an engine controller930, an operation panel940, and a network I/F950.

The controller910includes a CPU901as a main processor, a system memory (MEM-P)902, a north bridge (NB)903, a south bridge (SB)904, an application specific integrated circuit (ASIC)906, a local memory (MEM-C)907as a memory, a HDD controller908, and an HD909as a memory. The NB903and the ASIC906are connected through an accelerated graphics port (AGP) bus921.

The CPU901is a controller that controls entire operation of the image forming apparatus. The NB903connects the CPU901to the MEM-P902, the SB904, and the AGP bus921. The NB903includes a memory controller for controlling reading or writing of various data from or to the MEM-P902, a Peripheral Component Interconnect (PCI) master, and an AGP target.

The MEM-P902includes a ROM902a as a memory that stores a program and data for implementing various functions of the controller910. The MEM-P902further includes a RAM902bas a memory that deploys the program and data, or as a drawing memory that stores drawing data for printing. The program stored in the RAM902bmay be stored in any computer-readable recording medium, such as a compact disc-read only memory (CD-ROM), compact disc-recordable (CD-R), or digital versatile disc (DVD), in an installable or executable file format for distribution.

The SB904connects the NB903to a PCI device or a peripheral device. The ASIC906is an integrated circuit (IC) dedicated to an image processing use and including hardware elements for image processing, and connects the AGP bus921, a PCI bus922, the HDD controller908, and the MEM-C907to each other. The ASIC906includes a PCI target, an AGP master, an arbiter (ARB) as a central processor of the ASIC906, a memory controller, a plurality of direct memory access controllers (DMACs), and a PCI unit. The memory controller controls the MEM-C907. The DMACs is capable of rotating image data with a hardware logic. The PCI unit transfers data to a scanner controller931, a printer controller932, and a facsimile controller933through the PCI bus922. The ASIC906may be connected to a USB interface, or the Institute of Electrical and Electronics Engineers 1394 (IEEE 1394) interface.

The MEM-C907is a local memory used as a buffer for image data to be copied or a code buffer. The HD909stores various image data, font data for printing, and form data. The HDD controller908reads or writes various data from or to the HD909under control of the CPU901. The AGP bus921is a bus interface for a graphics accelerator card, which has been proposed to accelerate graphics processing. Through directly accessing the MEM-P902by high-throughput, the speed of the graphics accelerator card is improved.

The short-range communication circuit920is provided with a short-range communication circuit antenna920a. The short-range communication circuit920is a communication circuit in compliance with a standard such as near field communication (NFC) or Bluetooth (registered trademark).

The engine controller930includes the scanner controller931, the printer controller932, and the facsimile controller933. The operation panel940includes a panel display940aand a hard keypad940b. The panel display940ais implemented by, for example, a touch panel that displays current settings or a selection screen to receive a user input. The hard keypad940bincludes a numeric keypad that receives set values of various image forming parameters such as an image density parameter and a start key that receives an instruction for starting copying. The controller910controls overall operation of the image processing apparatus. For example, the controller910controls drawing, communication, or user inputs to the operation panel940. The scanner controller931and the printer controller932each performs various image processing, such as error diffusion or gamma conversion.

In response to an instruction to select a specific application through the operation panel940, for example, using a mode switch key, the image processing apparatus selectively performs a document box function, a copier function, a printer function, and a facsimile function. When the document box function is selected, the image processing apparatus operates in a document box mode. When the copier function is selected, the image processing apparatus operates in a copy mode. When the printer function is selected, the image processing apparatus operates in a printer mode. When the facsimile function is selected, the image processing apparatus operates in a facsimile mode.

The network I/F950controls communication of data with an external device through the network N2. The short-range communication circuit920and the network I/F950are electrically connected to the ASIC906through the PCI bus922.

Functions

Next, the functional configuration of the data output system100according to this embodiment will be described with reference toFIG.10.FIG.10illustrates an example functional configuration of the device20, the terminal apparatus30, and the information processing system10according to this embodiment.

Device

First, the device20includes a second communication unit21, a display control unit22, an operation acceptance unit23, an image data generation unit24, a facsimile processing unit25, an email processing unit26, and an output unit27. The second communication unit21, the display control unit22, the operation acceptance unit23, the image data generation unit24, the facsimile processing unit25, the email processing unit26, and the output unit27are functions or means implemented by the CPU901executing instructions included in one or more programs installed in the device20. For example, the second communication unit21, the display control unit22, and the operation acceptance unit23are implemented by a web browser, and the other components are implemented by individual applications (native applications).

The second communication unit21transmits and receives various types of information to and from the information processing system10. In this embodiment, the second communication unit21receives screen information and the like of an application list screen from the information processing system10, and transmits image data of a form, an application ID, and the like to the information processing system10.

The display control unit22interprets screen information of various screens and displays the screen information on the panel display940a. The operation acceptance unit23accepts various operations of the administrator 8 or the user on various screens displayed on the panel display940a.

When the application for which selection is accepted by the operation acceptance unit23is for generating image data, the image data generation unit24scans a document with the scanner controller931to generate image data. The image data generation unit24is a scanning function. The image data generation unit24converts the image data into a PDF file. The image date (such as JPEG, TIFF, or PNG) may remain in the original format without conversion by the image data generation unit24. The facsimile processing unit25performs processing related to reception and transmission of a facsimile message by the facsimile controller933. In response to receipt of a facsimile message, the facsimile processing unit25requests the information processing system10to execute an application associated in advance. The image data generation unit24may request the information processing system10to execute an application corresponding to the sender (fax number) of the facsimile message.

The email processing unit26performs processing related to transmission and reception of an email message. In response to receipt of an email message, the email processing unit26requests the information processing system10to execute an application associated in advance. The email processing unit26may request the information processing system10to execute an application corresponding to the sender (email address) of the email message.

The output unit27performs an image forming process to output a sheet with an image formed thereon. The image may be formed by either electrophotography or inkjet printing.

Terminal Apparatus

The terminal apparatus30includes a first communication unit31, a display control unit32, and an operation acceptance unit33. The first communication unit31, the display control unit32, and the operation acceptance unit33are functions or means implemented by the CPU501executing instructions included in one or more programs installed in the terminal apparatus30. The program may be a web browser or dedicated software for communicating with the information processing system10.

The first communication unit31transmits and receives various types of information to and from the information processing system10or the external service system40. In this embodiment, the first communication unit31receives various types of screen information and the like from the information processing system10or the external service system40, and transmits information set by the administrator 8 or the user to the information processing system10or the external service system40.

The display control unit32interprets screen information of various screens to display screens on the display506. The operation acceptance unit33accepts various operations of the administrator 8 or the user on various screens displayed on the display506.

Information Processing System

The information processing system10includes an application data service unit11, a setting acceptance unit12, an application data storage unit13, an authentication service unit14, a web service processing unit15, an input/output service unit16, an application processing information storage unit17, a component18, and a communication unit19. The application data service unit11, the setting acceptance unit12, the application data storage unit13, the authentication service unit14, the web service processing unit15, the input/output service unit16, the application processing information storage unit17, the component18, and the communication unit19are functions or means implemented by the CPU501executing instructions included in one or more programs installed in the information processing system10. InFIG.10, the information processing system10includes the function units described above for convenience of illustration. The function units may be arranged in a distributed manner in a plurality of information processing apparatuses.

The communication unit19transmits and receives various types of information to and from the terminal apparatus30and the device20. The setting acceptance unit12provides a setting site to the terminal apparatus30and accepts settings for the setting site. The term “site” may refer to a web page or a web application, or may refer to a system on the server side that provides a web page or a web application to the terminal apparatus30. The setting acceptance unit12includes a terminal screen providing unit42. The terminal screen providing unit42provides an initial screen that prompts the administrator 8 to set information to be used to execute an application, an application list screen, an association screen, and the like. In response to a Hypertext Transfer Protocol (HTTP) request from the terminal apparatus30, the terminal screen providing unit42generates screen information described in Hypertext Markup Language (HTML), Extensible Markup Language (XML), Cascade Style Sheet (CSS), JavaScript (registered trademark), or the like, and transmits the screen information to the terminal apparatus30as an HTTP response.

The application data service unit11includes an application data management unit41. The application data management unit41manages, for each customer such as company, application data43of applications purchased by the customer. For example, the application data43associates an application with the setting file described above. The application data43is stored in the application data storage unit13. An organization such as a customer viewed from the data output system100is referred to as “tenant”.

The web service processing unit15provides screen information to the device20or accepts execution of an application from the device20. The web service processing unit15includes an application execution unit46and a device screen providing unit47. The application execution unit46accepts execution of an application from the device20and requests an input/output processing unit48to execute the application. The device screen providing unit47uses screen information54to provide the device20with the screen information of an application list screen and an application screen.

The input/output service unit16inputs and outputs data by using the authentication service unit14, the application processing information storage unit17, and the component18. The input/output service unit16includes the input/output processing unit48and an application management unit49. The input/output processing unit48executes the application requested by the application execution unit46. When executing the application, the input/output processing unit48refers to application processing information53and requests the component18to perform processes included in the application.

The application management unit49manages the application processing information53of the application and the screen information54, which are stored in the application processing information storage unit17. The application processing information53includes settings indicating which processes and in which order the application is to perform the processes. For example, the application processing information53includes settings in which OCR is followed by making a request to the external service system40to perform processing. The screen information54is information on screens to be displayed on the device20, such as icons of applications, an application list screen, and application screens. The application processing information storage unit17stores the application processing information53and the screen information54for each application.

The authentication service unit14performs a process for authenticating a user (including the administrator8) and the device20. The authentication service unit14includes a user management unit44and a device management unit45. The user management unit44manages information related to the user. The authentication service unit14stores authentication information such as a user name and a password. The authentication information includes authentication information for the information processing system10and authentication information for the external service system40. The device management unit45manages the device20associated with the tenant. That is, the identification information of the tenant and the device20are managed in association with each other such that, in response to the device20being identified, the tenant is also identified.

The component18generally refers to individual functions that actually perform the respective processes. One block in the component18corresponds to one process. The component18includes, for example, an authentication processing unit61, an external service processing unit62, a determination unit63, a PDF processing unit64, an area detection unit65, a data storing unit66, an area identifying unit67, an OCR processing unit68, an OCR correction unit69, an OMR processing unit71, a trimming processing unit72, and a color detection unit73.

The authentication processing unit61is the component18that requests the authentication service unit14to perform a process related to authentication.

The external service processing unit62is the component18that requests the external service system40to perform a process. That is, the external service processing unit62is a function of cooperating with the external service system40. The external service processing unit62is prepared for each type of external service system40.

The determination unit63performs various determinations and controls branches of the process flow. The PDF processing unit64performs processes related to a PDF file. For example, the PDF processing unit64extracts image data from the PDF file or creates a PDF file with transparent text. This is because the device20transmits image data in the format of a PDF file.

The area detection unit65detects a mark associated with a process from image data of a registration sheet. Further, the area detection unit65detects a rectangular area marked with a marker pen or the like from image data of a form and creates the coordinates (area information) of the rectangular area.

The area identifying unit67identifies a processing-target area from the image data of the form, based on the area information. The area identifying unit67acquires (for example, clips) the identified processing-target area. The OCR processing unit68performs OCR processing on an area and the entire image data. The OCR correction unit69corrects an OCR processing result. The OMR processing unit71performs OMR processing on an area. The trimming processing unit72performs trimming processing on an area. The component18may further include components such as a translator in addition to the illustrated ones.

The data storing unit66registers a color and a process in association with each other, and further registers the process and area information in association with each other in the setting file. The data storing unit66outputs a file including acquired data generated by the OCR processing unit68, the OMR processing unit71, and the trimming processing unit72through processing. The acquired data is data acquired from an area of the form.

The color detection unit73detects a color of the registration sheet that is colored by the administrator 8 using a marker pen.

External Service System

The external service system40stores a registration sheet 5 and a setting file 9. The registration sheet 5 is registered in advance. The setting file 9 is registered by the administrator 8.

Table 1 presents an example of application processing information stored in the application processing information storage unit17. The application processing information storage unit17stores a list of applications that have been purchased for (or are licensed by) each tenant. The information presented in Table 1 is a list of applications for performing image processing on a portion of received image data, which will be described in this embodiment, among the applications included in the list stored in the application processing information storage unit17, and includes the following items: application ID, application name, screen information, application processing information, shared application, and external service system. The application ID is identification information identifying an application. The application name is the name of an application assigned by the administrator 8. An application is associated with the screen information54(described in HTML or the like) to be used when the icon of the application or a setting screen is displayed on the device20. Since the content and order of a series of processes (process flow) are defined for an application, each application is associated with the application process information53. As described below, a copied application has the same flow information as that of the source application. The shared application is an application available to users of the tenant in common, whereas a typical application is an application that each user is authorized to use. In Table 1, an expense claim form application is a shared application. The expense claim form application is used by users in common. The external service system is an external service system40with which each application cooperates. Some applications do not cooperate with any external service system40.

When the administrator 8 or the user logs in, the application management unit49acquires a list of applications that the administrator 8 or the user is authorized to use from the application processing information storage unit17, and provides the list to the device20. When an application is designated from the list of applications, the application management unit49identifies screen information and application processing information associated with the application.

Table 2 presents an example of information related to users managed by the user management unit44. Table 2(a) includes the following items of information related to a user in the information processing system10: user ID, user name, email address, password, available application, and external service system information. The user also includes the administrator 8. The password and at least one of the user ID, the user name, or the email address are authentication information for the user to log in to the information processing system10. The available application is an application that the user is authorized to use. Each user is authorized to use an application in response to the administrator 8 assigning permissions for the user to access the application for which the administrator 8 has purchased a license while designating the number of users or the like. The available application includes a personal application and a shared application. Both the personal application and the shared application are used by a plurality of users. The personal application can be accessed by an individual user using their authentication information for an external service, whereas the shared application can be accessed by a plurality of users using authentication information of the administrator 8 for the external service. In Table 2(a), app003, which is common to U001 to U003, is a shared application. The other applications are personal applications. The personal application involves registering authentication information of each user for an external service system. The external service system information is the external service system40with which the user has created a setting for cooperation.

The user has created a setting for cooperation with an external service system40upon it being confirmed that the user is allowed to log in with authentication information for the external service system40with which the user desires to cooperate. The user enters authentication information for the external service system40from the setting site. Upon successful login, the identification information for the external service system40is set in the external service system information in Table 2(a). Authentication information for the external service system40is registered in Table 2(b).

Table 2(b) represents information related to the external service system40for each user, and includes the following items: user ID, external service system, user ID, and password. Authentication information for the external service system40with which the user has created a setting for cooperation is registered in Table 2(b). The authentication information for the external service system40is authentication information for the user to log in to the external service system40. Some users, such as a user with U002, may have different authentication information for each external service system40. As a result, each user of the data output system100is associated with authentication information for the external service system40. The information processing system10identifies the authentication information for the external service system40with which a setting for cooperation has been created, in accordance with authentication information for the information processing system10acquired from the device20, such as the user ID and the password.

The authentication information is authentication information for the external service system40or authentication information to be used to cooperate with the external service system40, and may be, instead of a user ID or a password, a token or the like acquired from the external service system40at the time of setting for cooperation with the external service system40.

Table 2(c) presents authentication information for the shared application. Identification information for the shared application is associated with the external service system40and authentication information. Since the authentication information of the administrator 8 is used for the shared application, the identification information of the administrator 8 is registered as the identification information for the shared application.

Table 3 presents an example of the application data43stored in the application data storage unit13. The application data43associates an application with a setting file (area information). An association between an application and a setting file is performed by the administrator 8. The “applicationId” is the application ID of an application that cooperates with a cooperation application. The “applicationName” is the application name of the application. The “fileId” is the identification information of a setting file created by the information processing system10. The “fileName” is the file name of a setting file created by the information processing system10. The setting file is stored in the external service system40, and the file name of the setting file is set together with the folder of the setting file. The storage location may be identified by, for example, uniform resource locator (URL). The “Upload destination folder” is the external service system40and a folder in which the setting file is stored. The “Upload destination folder” may include, for example, a URL. The administrator 8 sets an upload destination folder.

Table 4 presents an example of the setting file 9 stored in the external service system40. The setting file 9 is a file including area information to be subjected to OCR processing. The setting file 9 is in a tabular format (e.g., a CSV format), and has cells whose locations are each designated by a column number and a row number. The items in columns A to M will be described hereinafter. The item names in the first row of the setting file 9 are automatically set by the information processing system10upon creation of the setting file9.

“ID” is identification information of area information. The IDs are automatically numbered without the administrator 8 or the user having to pay attention to them.

“Marker color” is the color of a cell (illustrated inFIG.13) read from the registration sheet (an example of identification information identifying the content of a process). The color of the cell is registered in, for example, the HSV color space. In the HSV color space, “H” refers to hue and indicates a “color”. The range of “H” is 0 to 360 degrees. “S” refers to saturation and indicates a level of “vividness”. The range of “S” is 0 to 100%. “V” refers to value and indicates a level of “brightness”. The range of “V” is 0 to 100%. The color space representing the marker color may be HSB for hue, saturation, brightness, RGB for red, green, and blue, Lab, XYZ, or the like.

“Process” is the content of a process associated with a cell in the registration sheet. Examples of the process include OCR, OMR, and trimming. Other examples may include translation, masking (filling), and encryption.

“Start point (X coordinate)”, “start point (Y coordinate)”, “end point (X coordinate)”, and “end point (Y coordinate)” are included in area information. These points represent the diagonal vertices of a marked rectangular area of the form. The coordinates, width, and height of one point may be designated. The coordinates may be expressed in units of cm, pixels, or the like, for example.

The information described above is automatically set by the information processing system10. The following information is entered by the administrator 8.

“Format” is a data format of the area. Examples of the data format include text, Y/N, image, date (YYYY/MM/DD), and numerical value (NUMERIC). Since the data format is specified, the OCR processing unit68can perform OCR processing with higher accuracy. The format can be selected from a pull-down menu. As described above, a data type is set for each area.

“CSV column name” is a name assigned to the area. In the case of a form, for example, the title of a rectangular frame is input. The administrator 8 can enter any value. The CSV column name (if the CSV position is designated) is output to a file together with the data.

“CSV position (column)” designates in which column of the file the information processing system10writes the data extracted from the area.

“CSV position (start row)” designates in which row of the file the information processing system10writes data extracted from the area. The row number increases with each reading of the form, and thus the “start row” when no data is described in the file is set. For example, data acquired from the first form is set in the second row, and data acquired from the second form is set in the third row. In contrast, the column positions may be fixed.

“OCR language” indicates the language in which data is described in the area. Since the language is specified, the OCR processing unit68can perform OCR processing with higher accuracy. The OCR language can be selected from a pull-down menu. As described above, a data type is set for each area.

“Order of use for file name” is used for the file name of a file to which data acquired from the area is output in this embodiment. The order of use for a file name is the order of data to be used for the file name of a file to be output from the information processing system10. For example, if the “document name” is assigned the first place in the order, the file name is “expense claim form.csv” in accordance with the data (expense claim form) acquired from the “document name” area.

Application Processing Information

Next, the application process information will be described in detail with reference toFIG.11.FIG.11illustrates an example of the application processing information53stored in the application processing information storage unit17.

The application processing information53illustrated inFIG.11is information related to a series of processes for implementing a service for requesting the “external service system B” to process image data generated by the device20through scanning.

The application process information53includes a flow ID1201A indicating identification information of the application process information53, a flow name1202A indicating the name of the application process information53, and flow details1203A indicating the content of the series of processes. The flow name1202A is also the name of the application.

Further, the flow details1203A include a process content1223A indicating that a request is made to the “external service system B” to perform processing.

The process content1223A specifies the component ID “ExService_B” of the component18that executes a process in cooperation with the “external service system B”, and the parameters “AppId” and “Folder”. The “ExService_B” corresponds to the external service processing unit62for the external service system B. The “AppId” stores the cooperation application ID. The correspondence between the application and the cooperation application is determined in advance. The “Folder” stores the folder for storing the file, i.e., the upload destination folder. The upload destination folder set by the administrator 8 is set as the folder in the storage location. As a result, the information processing system10can store the scanned image data in the “external service system B”.

In the case of an application having no cooperation application, “AppId” is omitted. The illustrated parameters are some of parameters that may be included in the application processing information.

While a single component is designated inFIG.11for simplicity of illustration, a series of process contents1223A specifying processes to be performed by the application, the number of which is equal to the number of processes (components), are registered in the application process information53.

FIGS.12A to12Cschematically illustrate series of processes performed by applications.FIG.12Aillustrates a series of processes of an application (an application corresponding to a color registration button112described below) for registering a correspondence between a marker color and a process in a setting file. One block corresponds to a process content1223A. The details of the processes will be described with reference toFIG.15.

FIG.12Billustrates a series of processes of an application (an application corresponding to a form registration button113described below) for registering area information in a marker color and a process in a setting file. The details of the processes will be described with reference toFIG.25.

FIG.12Cillustrates a series of processes of an application (e.g., an expense claim form application) for outputting data from an area of a form. The details of the processes will be described with reference toFIGS.37A and37B.

Registration of Correspondence between Color and Process in Setting File

Next, setting of a correspondence between a color and a process by the administrator 8 and creation of a setting file will be described with reference toFIGS.13to20.

FIG.13illustrates a method for setting a correspondence between a color and a process using a registration sheet.FIG.13(a)illustrates a registration sheet before the administrator 8 colors the registration sheet with marker pens. The original of the registration sheet is registered in the external service system40.

The first row of the registration sheet specifies items of the respective columns (marker color in the first column, and process in the second column). This embodiment includes three “processes” (OCR, OMR, and trimming), and the processes are set in the registration sheet in advance in the following order: OCR in the second row; OMR in the third row; and trimming in the fourth row. In the original of the registration sheet, cells130designating marker colors are blank.

FIG.13(b)is a view in which the administrator 8 colors the cells130with marker pens for the respective processes. For example, the administrator 8 uses green, blue, and red marker pens among the owned marker pens. The administrator 8 assigns green to OCR processing, blue to OMR processing, and red to trimming processing. The original of the registration sheet includes predetermined spacings between rows to avoid mixture of color inks due to spreading of the color inks.

The administrator 8 colors each of the cells130of the registration sheet with a marker pen among the owned marker pens in accordance with the assignment of the desired marker color to the process, which is determined by the administrator 8. The device20scans the registration sheet to register the correspondence between the marker colors and the processes in the information processing system10.

While the cells130are filled in with colors inFIG.13, each of the cells130may be marked with any mark. For example, the cells130may be marked with signs such as triangles or circles, alphabets, borders, or the like. In this case, the information processing system10registers such marks and the processes in the setting file. The information processing system10may or may not consider the colors of the marks.

Instead of filling in the cells130with colors using marker pens, the administrator 8 may enter text or signs indicating the respective marker colors, such as “green or G”, “blue or B”, or “red or R”. In this case, the information processing system10performs character recognition to associate a marker color with the content of a process.

Alternatively, the administrator 8 may connect the terminal apparatus30to the information processing system10, and may enter text or the like to a web page displayed on the terminal apparatus30to specify a color.

In addition, the administrator 8 may be allowed to register the registration sheet itself. The administrator 8 connects the terminal apparatus30to the information processing system10, and registers the content of processes in a web page displayed on the terminal apparatus30to create a registration sheet illustrated inFIG.13(a).

Screen Example on Device

FIG.14illustrates an example of an administrator screen110displayed in response to the administrator 8 logging in to the device20. The administrator screen110includes an administrator menu118and a general user menu119. The administrator menu118is a menu for the administrator 8 to register the setting file. The general user menu119is a list of applications available to the administrator 8 as a user.

In the administrator menu118, a registration sheet button111is a button for the administrator 8 to print a registration sheet. A color registration button112is a button for the administrator 8 to scan a registration sheet with the device20and register a correspondence between marker colors and processes in the setting file. The form registration button113is a button for the administrator 8 to scan a marked form with the device20and register a correspondence between processes and area information in the setting file.

The registration sheet button111, the color registration button112, and the form registration button113are web applications. The administrator 8 performs tasks in accordance with the respective web applications to register the setting file.

In response to pressing of the registration sheet button111of the administrator menu118, the device20displays a registration sheet. The administrator 8 prints the registration sheet and colors the registration sheet with marker pens. Then, in response to the administrator 8 pressing the color registration button112of the administrator menu118and executing scanning of the registration sheet colored using the marker pens, image data of the registration sheet is transmitted to the information processing system10. The information processing system10registers the correspondence between the marker colors and the processes in the setting file.

Then, the administrator 8 presses the form registration button113of the administrator menu118, and the device20scans the original of the form colored using the marker pens. Image data of the form is transmitted to the information processing system10, and the information processing system10registers the correspondence between the processes and the area information in the setting file.

In response to pressing of the form registration button113, the device20may display the correspondence between the marker colors and the processes, which is received from the information processing system10. For example, the device20displays the information illustrated inFIG.13(b)on the operation panel940. To change the correspondence between the colors and the processes, the administrator 8 presses an up/down button114on the operation panel940. Each time the up/down button114is pressed, the device20shifts one row in the column for the process. This enables the administrator 8 to change the correspondence between the colors and the processes.

In response to the administrator 8 pressing, as a user, an application in the general user menu119, the application is activated. In response to the user scanning the filled in form with the device20, the information processing system10executes, on each area, a process associated with the area identified by the area information in the setting file.

FIG.15is a sequence chart illustrating an example procedure for the administrator 8 to register a correspondence between a color and a process in a setting file using a registration sheet.

S1: The administrator 8 operates the device20to log in to the information processing system10, and presses the registration sheet button111in the administrator menu118.

S2: The second communication unit21of the device20transmits a request to the information processing system10to execute an application corresponding to the registration sheet button

S3: The communication unit19of the information processing system10receives the request for executing the application. The application execution unit46designates the application ID of the application corresponding to the registration sheet button111and requests the input/output processing unit48to execute the application. The input/output processing unit48refers to the application processing information53associated with the application ID of the application corresponding to the registration sheet button111and executes the application. First, the input/output processing unit48requests the external service processing unit62to provide a list of registration sheets.

S4: The external service processing unit62communicates with the external service system40using the authentication information of the administrator 8 and requests the external service system40to provide a list of registration sheets available to the administrator 8.

S5: The second communication unit21of the device20receives the list of registration sheets, and the display control unit22displays the list of registration sheets. The administrator 8 selects a registration sheet in which a process to be executed is registered. The operation acceptance unit23accepts the selection of a registration sheet. If a single registration sheet is included, the user provides a printing instruction instead of selection.

S6: The output unit27of the device20prints the registration sheet selected by the administrator 8. In the sequence chart, a process for the device20to acquire the registration sheet from the external service system40is omitted. On the registration sheet, identification information of the registration sheet or position information of the cells130to be colored may be provided in form of barcode or the like. The details will be described with reference toFIGS.16A and16B.

S7: The administrator 8 colors the cells130associated with processes in the printed registration sheet with marker pens. Then, the administrator 8 presses the color registration button112in the administrator menu118.

S8: The operation acceptance unit23of the device20accepts pressing of the color registration button112, and the image data generation unit24scans the registration sheet to generate a PDF file.

S9: The second communication unit21of the device20transmits the PDF file and a request for executing the application corresponding to the color registration button112to the information processing system10.

S10: The communication unit19of the information processing system10receives the PDF file. The application execution unit46designates the application ID of the application corresponding to the color registration button112and requests the input/output processing unit48to execute the application. The input/output processing unit48refers to the application processing information53associated with the application ID of the application corresponding to the color registration button112and executes the application. First, the input/output processing unit48designates the PDF file and requests the PDF processing unit64to extract image data. The PDF processing unit64extracts the image data of the registration sheet from the PDF file.

S11: Then, the input/output processing unit48transmits the image data to the color detection unit73to make a request to detect the correspondence between the colors and the processes.

S12: The color detection unit73detects color information from the cells130determined in advance in the image data.

FIGS.16A and16Bschematically illustrate detection of color information. The registration sheet has a barcode101at a corner thereof such as the upper left corner, and the barcode101indicates the orientation of the registration sheet. The color detection unit73rotates the image data such that the barcode101is located at the upper left corner. Then, the color detection unit73detects color information from the cells130determined (in position) in advance for the respective processes.

As illustrated inFIG.16A, the process of OCR is determined by the fill color of a rectangular cell defined by two position coordinate points A and B (an example of position information) indicating diagonal corners. The process of OMR is determined by the fill color of a rectangular cell defined by position coordinate points C and D. The process of trimming is determined by the fill color of a rectangular cell defined by position coordinate points E and F. The correspondence between the sets of position coordinate points A and B, position coordinate points C and D, and position coordinate points E and F and the respective processes may be stored in the barcode101or in the application data management unit41. Also in this case, the barcode101may include at least identification information of a registration sheet; the identification information may be omitted if a single type of registration sheet is present.

As illustrated inFIG.16B, the barcode101or the application data management unit41stores the processes and the sets of position coordinate points of the respective cells130to be colored in association with each other. The color detection unit73acquires the H value of the HSV color system from each of the cells130, based on the position coordinate points illustrated inFIG.16B. Upon detection of the color (H=(0 to 60°, 301 to 360°)) corresponding to red from the cell130associated with trimming, the color detection unit73registers H=(0 to 60°, 301 to 360°) of the HSV color system in the setting file in association with trimming. Upon detection of the color (H=(61 to 180°)) corresponding to green from the cell130associated with OCR, the color detection unit73registers H=(61 to 180°) in the setting file in association with OCR. Upon detection of the color (H=(181 to 300°)) corresponding to blue from the cell130associated with OMR, the color detection unit73registers H=(181 to 300°) in the setting file in association with OMR. In this manner, instead of registering the entire ranges of the H values of red, blue, and green, the color detection unit73may register, in the setting file, colors (the H values) obtained by adding a margin such as ±10% to the detected H values.

The colors and the processes may be associated with each other by the administrator 8 using the management site150illustrated inFIG.4. The sequence of the setting using the management site150is as follows. The communication unit19transmits screen information of the management site150to the terminal apparatus30, and receives the correspondence between the marks and the processes, which is accepted by the terminal apparatus30. The data storing unit66stores the processes and the marks in the setting file in association with each other.

Referring back toFIG.15, the operation will still be described. S13: The application execution unit46acquires the correspondence between the colors and the processes from the color detection unit73. The application execution unit46transmits the correspondence between the colors and the processes to the data storing unit66, and makes a request to generate a setting file. The data storing unit66numbers IDs and creates a setting file in which the marker colors and the processes are associated with each other. The data storing unit66transmits the setting file to the application execution unit46. At this time, Table 4 contains information in the columns A to C. The correspondence between the marker colors and the processes may be stored not in the setting file but in a separate independent file.

S14, S15: The application execution unit46transmits the setting file to the external service system40via the external service processing unit62. The external service processing unit62acquires, from the user management unit44, authentication information of the administrator 8 for the external service system40. As a result, the external service processing unit62logs in to the external service system40, and stores the setting file in the top folder of the administrator 8 (or a folder designated by the administrator 8). If the administrator 8 designates the file name of the setting file, the external service processing unit62assigns the file name to the setting file. If no file name is set, the current date and time or the like is used as the file name.

As described above, the setting file in which the marker colors and the processes are associated with each other is registered in the external service system40.

Designation of File Name using Process

In this embodiment, the administrator 8 sets the file name of a CSV file in which the acquired data is to be stored. The administrator 8 may set the file name of the CSV file by coloring the registration sheet.

FIG.17illustrates an example of use of a registration sheet in which a process is applied to the setting of a file name. InFIG.17, the process is limited to OCR, and the data storing unit66is enabled to use a plurality of pieces of acquired data for file names. The administrator 8 colors the registration sheet with colors of marker pens to be designated for the respective file names. In the registration sheet illustrated inFIG.17(a), OCR is set as a process, and the order of use for the file names, that is, “File name 1” and “File name 2”, is set.

As illustrated inFIG.17(b), the administrator 8 uses marker pens to designate the color (e.g., green) of the marker pen to be used for File name 1 and the color (e.g., blue) of the marker pen to be used for File name 2.

As illustrated inFIG.17(c), a numerical value indicating a color and a process (to be used for a file name) are associated with each other in the setting file. Two methods are conceivable for associating colors with processes.

In a first method, information indicating whether to associate the coordinates (in the registration sheet) at which a color is acquired and the process obtained from the coordinates is stored in the information processing system10in advance. The color extracted at the coordinates (X coordinates: 100, 200; Y coordinates: 1, 50) is associated with file name 1. The color extracted at the coordinates (X coordinates: 100, 200; Y coordinates: 51, 100) is associated with file name 2.

In a second method, the administrator 8 associates colors with processes on a management site.FIG.18illustrates an example of a management site152for associating a color with a process. The administrator 8 can use radio buttons153to register colors to be used to designate file names. In the illustrated example, the user uses three marker pen colors.

In either method, as illustrated inFIG.17(c), a numerical value identifying the marker color (green) and File name 1 are associated with each other. A numerical value identifying the marker color (blue) and File name 2 are associated with each other. File names 1 and 2 indicate the order of use for the file names.

FIGS.19A and19Billustrate an example of a form in which the administrator 8 designates two areas using marker pens.FIG.19Aillustrates the original of the form, andFIG.19Billustrates the form colored with marker pens. The device20scans the form illustrated in FIG.19B, and the information processing system10identifies the colored areas in the form and registers area information in the setting file.

InFIG.19B, the “expense claim form” area, which is the title of the form, is designated by a green marker pen, and the area corresponding to the application date is designated by a blue marker pen. For example, when the application date is “2020/10/10”, the file name of the CSV file obtained when the form illustrated inFIG.19Bis scanned by the device20is “expense claim form20201010.csv”. Three or more marker pen colors may be used to specify file names.

The order of areas to be used for file names (file 1, file 2) may not necessarily be registered. In this case, the data storing unit66sets the file names in the order of registration of the colors in the setting file. As a result, colors of marker pens are not changed from cell to cell of the registration sheet.

Modification of Registration Sheet

As illustrated inFIG.20, a CSV position may be registered in advance in the registration sheet.FIG.20illustrates a modification of the registration sheet.FIG.20(a)illustrates the registration sheet before coloring is applied. The registration sheet includes columns of a CSV column name154and a CSV position155, and the administrator 8 does not manually set a CSV position in the setting file.

FIG.20(b)illustrates the registration sheet after coloring is applied. The registration sheet illustrated inFIG.20(b)is scanned by the device20.

FIG.20(c)illustrates a portion of a setting file in which colors, processes, and area information are associated with each other.

Registration of Area Information

Next, registration of the area information will be described with reference toFIGS.21A to21Cand the like.FIGS.21A to21Cillustrate a method for marking areas in a form.FIG.21Aillustrates the form (original) before areas are designated using marker pens. The administrator 8 designates, on the original of the form, an area on which a process registered in the registration sheet is to be executed, using a marker pen of a registered color. For example, the administrator 8 is to set OCR for the title area140, OMR for the preliminary application area141, and trimming for the remarks area142.

FIG.21Billustrates a form sheet on which areas are colored with colors of marker pens. The title area140representing “expense claim form”, which is the title of the form, is colored with a green marker pen. The preliminary application area141is colored with a blue marker pen.

The remarks area142located in the lower portion of the form is colored with a red marker pen. The device20scans the form sheet to register the relationship between the areas of the form and the processes in the setting file.

According to the setting file presented in Table 4, the OCR processing unit68performs OCR processing on the title area140representing “expense claim form”, which is the title of the form, to generate acquired data. The OMR processing unit71performs OMR processing on the preliminary application area141to generate acquired data. The trimming processing unit72performs trimming on the remarks area142to generate acquired data (e.g., “I apologize for late application”). The acquired data obtained by trimming is image data (such as JPEG, Graphic Interchange Format (GIF), PNG, TIFF, or BMP data).

FIG.21Cillustrates an example of a form with items filled in by the user. From the colored areas illustrated inFIG.21B, “expense claim form”, “preliminary application applicable”, and “I apologize for late application” are acquired data.

As illustrated inFIG.22A, the areas may not necessarily be filled in with colors. InFIG.22A, circle lines602surrounding areas are drawn with a marker pen. The area detection unit65detects pixels other than black and white and detects rectangles circumscribing the pixels other than black and white (the circle lines602).

The method for marking areas is not limited to the method illustrated inFIGS.21A to21C or22A and22B. For example, an area may be designated by a double line or an area may be designated within a range surrounded by a predetermined symbol such as %, $, or # or by a predetermined shape such as a triangle.

As illustrated inFIG.22B, one form may include a plurality of areas colored with the same color. For example, as illustrated inFIG.22B, in response to the administrator 8 designating areas143of data in the form using a green marker pen, all of the areas143are subjected to OCR processing.

Then, the administrator 8 presses the form registration button113illustrated inFIG.14. The administrator 8 logs in to the information processing system10from the device20and selects the form registration button113.

The effect of the administrator 8 filling in an area with a color to designate the area will be described with reference toFIGS.23A and23B.FIGS.23A and23Billustrate comparative examples of a method for designating an area in a form.

FIG.23Aillustrates a designation method in which the administrator 8 colors an area with a marker pen and adds a triangle mark161to the colored area by handwriting. This method, however, is a waste of time.

FIG.23Billustrates a designation method in which the administrator 8 handwrites a triangle mark162so as to surround an area using a marker pen. This method, however, fails to completely determine an area and involves complicated processing for recognizing the area. For example, after recognizing the triangle mark162, the information processing system10recognizes the area indicated by the rectangle.

In contrast, coloring (filling in) an area with a marking pen enables the administrator 8 to perform “designation of an area” and “designation of a process” at the same time in a simple way.

Designation of an area: a shape, such as a rectangle

Designation of a process: a feature other than shapes, such as a color

Area Registration Screen

FIG.24illustrates an example of an area registration screen610displayed on the device20upon registration of area information. The area registration screen610includes a message611indicating “Set a form with a colored area to be processed and then press ‘start’”, a reading settings field612, and a start button613. The user sets a document (form) on the contact glass and presses the start button613. The user may set reading conditions, such as color/monochrome, simplex/duplex, orientation, and resolution, in the reading settings field612, as appropriate. The color/monochrome setting may be fixed to color to prevent the user from erroneously setting monochrome.

The administrator 8 may be able to designate the file name of the setting file from the area registration screen610illustrated inFIG.24. This facilitates searching for the setting file from the external service system40. In addition, the administrator 8 may be able to set the folder for storing the setting file.

FIG.25is a sequence chart illustrating an example procedure for the information processing system10to acquire area information from a form with an area marked by the administrator 8 and create a setting file.

S101: The administrator 8 operates the device20to log in to the information processing system10, and presses the form registration button113in the administrator menu118. Accordingly, the display control unit22of the device20displays the area registration screen610. The administrator 8 places the form with the marked area on the contact glass and presses the start button613. The image data generation unit24generates image data of the form and converts the image data into a PDF file.

S102: The second communication unit21of the device20transmits the PDF file to the information processing system10together with a request for executing the application corresponding to the form registration button113.

S103: The communication unit19of the information processing system10receives the PDF file. The application execution unit46designates the application ID of the application corresponding to the form registration button113and requests the input/output processing unit48to execute the application. The input/output processing unit48refers to the application processing information53associated with the application ID of the application corresponding to the form registration button113and executes the application. First, the input/output processing unit48designates the PDF file and requests the PDF processing unit64to extract image data. The PDF processing unit64extracts the image data of the form from the PDF file.

S104: To determine a color to be extracted from the image data, the application execution unit46requests the external service processing unit62to provide a setting file.

S105: The external service processing unit62acquires, from the external service system40, a setting file that can be acquired using the authentication information of the administrator 8. The administrator 8 may select a desired setting file from a list of setting files.

S106: Then, the input/output processing unit48designates a marker color registered in the image data and the setting file and requests the area detection unit65to acquire an area. The area detection unit65acquires an area from the image data, based on the color of the image data.

FIGS.26A to26Cillustrate detection of an area from the image data. As illustrated inFIG.26A, a colored portion601designated with a marker pen has a color corresponding to the marker pen. For example, in the HSV color space, the range of red is H=(0 to 60° , 301 to 360°), the range of green is H=(61 to 180°), and the range of blue is H=(181 to 300°).

For example, the area detection unit65searches the image data for a color included in the range H of the marker color in the setting file. The area detection unit65detects only a continuous colored area having an area greater than or equal to a certain value. This prevents detection of a portion erroneously colored by the user.

FIG.26Billustrates the detected colored portion601. The area detection unit65sets a rectangle620circumscribing the detected colored portion601. Then, the area detection unit65sets margins on the upper, lower, left, and right sides of the circumscribing rectangle620and determines an area621including the margins. InFIG.26B, margins of a (in mm) are set to the left and right, and margins of b (in mm) are set to the top and bottom. The margins are set to avoid a position shift or the like caused when the device20reads a form. InFIG.26B, the coordinates of an upper left corner622are the start point X coordinate and the start point Y coordinate of the area information, and the coordinates of a lower right corner623are the end point X coordinate and the end point Y coordinate of the area information.

As illustrated inFIG.26C, the area detection unit65may detect a box line625outside the colored portion601detected based on the range H of the HSV color space. The area detection unit65searches for black pixels toward the outside of the colored portion601(the circumscribing rectangle620) in the vertical direction, traces black pixels found, and determines that the box line625outside the colored portion601has been detected in response to the initially found black pixel being reached. This enables detection of the largest area in which data is to be written, and enables the area identifying unit67to easily identify the data even if a position shift or the like occurs when the device20reads a form.

The administrator 8 can set, for example, how much margin the area detection unit65provides for an area to be identified with a marker pen. The details will be described with reference to an association screen640-2illustrated inFIG.31. The area detection unit65sets area information based on the settings of the association screen640-2.

Referring back toFIG.25, the operation will still be described. S107: Then, the input/output processing unit48designates the image data for which the area is designated with a marker pen, and requests the external service processing unit62to upload the image data to the external service system40.

S108: The external service processing unit62acquires, from the user management unit44, authentication information of the administrator 8 for the external service system40. As a result, the external service processing unit62logs in to the external service system40, and stores the image data in the top folder of the administrator 8 (or a folder designated by the administrator8). If the administrator 8 designates the file name of the setting file on the area registration screen610, the external service processing unit62assigns the same file name to the file of the image data. If no file name is set, the current date and time or the like is used as the file name.

S109: Then, the input/output processing unit48designates the area information detected by the area detection unit65and requests the data storing unit66to create a setting file. The data storing unit66arranges the area information in a determined column of a CSV file to generate a setting file. At this time, Table 4 contains information in the columns A to G.

S110: Then, the input/output processing unit48designates the setting file and requests the external service processing unit62to upload the setting file to the external service system40.

S111: The external service processing unit62, which has logged in to the external service system40, stores the setting file (by overwriting old data) in the top folder of the administrator 8 (or a folder designated by the administrator8). Since the image data and the setting file have the same file name, the administrator 8 can easily determine from which image data the setting file is generated.

Registration of Other Information in Setting File

The setting of the columns H to M of the setting file (Table4) will be described with reference toFIGS.27and28.FIG.27is a sequence chart illustrating an example procedure for the administrator 8 to set information in the columns H to M of the setting file.

S31: After registering the setting file in the external service system40, the administrator 8 operates the terminal apparatus30and connects the terminal apparatus30to the external service system40. The administrator 8 inputs their authentication information and logs in to the external service system40.

S32: The external service system40, which is a storage service, transmits a list of files registered by the administrator 8 to the terminal apparatus30. For example, a list of files in the top folder is transmitted. If no file is contained in the top folder, the administrator 8 designates a hierarchy. The list of files also includes the setting file and the file of the image data registered by the administrator 8 from the device20.

S33: The administrator 8 selects a setting file to check the content of the setting file or set information in the columns H to M. The operation acceptance unit33of the terminal apparatus30accepts the selection, and the first communication unit31designates the file ID of the setting file and sends a request for the setting file to the external service system40.

S34: The communication unit19of the external service system40receives the request for the setting file and transmits the setting file to the terminal apparatus30.

S35: The administrator 8 checks the content of the setting file and sets information in the columns H to M. The operation acceptance unit33of the terminal apparatus30accepts the settings.

S36: For example, in response to the administrator 8 performing an operation of uploading the setting file, the first communication unit31transmits the setting file in which information is set in the columns H to M to the information processing system10.

As described above, the setting file is registered in the external service system40.

FIG.28illustrates a correspondence between a setting file and a form. As described in Table4, the administrator 8 sets the format, the CSV column name, the CSV position (column), the CSV position (row), the OCR language, and the order of use for a file name. Not all of the columns may be set by the administrator 8. The designation of the columns H to M is ignored depending on the content of the process in the column C.

In the example illustrated inFIG.28, the data acquired from the area information with ID=1 corresponds to “document name” in the CSV column name. The item “document name” is used first in the file name. The data acquired from the area information with ID=2 corresponds to “preliminary application” in the CSV column name. Since the CSV position indicating the preliminary application is set to column 3 and row 2, the data acquired from the area information with ID=2 is written to the third column and second row of the file. The data acquired from the area information with ID=3 corresponds to “remarks” in the CSV column name. Since the CSV position indicating the remarks is set to column 4 and row 2, the data acquired from the area information with ID=3 is written to the fourth column and second row of the file.

Association between Application and Setting File

Next, the association between an application and a setting file will be described with reference toFIGS.29and30.FIG.29illustrates an example of screens displayed on the terminal apparatus30for associating an application and a setting file with each other. The administrator 8 operates the terminal apparatus30and causes the terminal apparatus30to communicate with the information processing system10.

FIG.29(a)illustrates an initial screen630displayed in response to the administrator 8 logging in to the information processing system10. An operation menu permitted for the administrator8is displayed. To associate an application with a setting file, the administrator 8 selects an icon631for displaying a list of applications.

As presented in Table 2, since applications available to each user and external service system information are registered in the user management unit44, the information processing system10can provide, to the terminal apparatus30, a list of applications available to the administrator 8 for each external service system40.

FIG.29(b)illustrates an example of an application list screen632. The application list screen632displays a list of applications available to the administrator 8. The administrator 8 selects, from the application list screen632, an application633(e.g., an expense claim form application) to be associated with a setting file 9 registered in the external service system40. In response to the operation acceptance unit33accepting the selection of the application633, an association screen640illustrated inFIG.29(c)is displayed. The association screen640includes an external service system selection field641, an upload destination folder field642, and a setting file field643.

The external service system selection field641is a field for the administrator 8 to set the external service system40as the storage location of a file to which data acquired from the area is to be written. Accordingly, the administrator 8 can change the external service system40associated with the application in advance or set a new external service system40.

The upload destination folder field642is a field for the administrator 8 to set a folder for storing the file to which data acquired from the area is to be written in the external service system40. In response to the administrator 8 pressing a folder selection button642a, the information processing system10logs in to the external service system40using the authentication information of the administrator 8 for the external service system40, and the top folder accessible to the administrator 8 in the external service system40is displayed on the terminal apparatus30. The administrator 8 can select any folder.

The setting file field643is a field for the administrator 8 to select a setting file to be associated with the application. In response to the administrator 8 pressing a file selection button644, the information processing system10logs in to the external service system40using the authentication information of the administrator 8 for the external service system40, and the top folder accessible to the administrator 8 in the external service system40is displayed on the terminal apparatus30. The top folder (or a folder set by the administrator8) includes a setting file registered by the information processing system10and updated by the administrator 8. If the top folder does not include the setting file, the administrator 8 can search for the setting file from a folder below the top folder.

FIG.30is a sequence chart illustrating an example process for the administrator 8 to associate an application with a setting file via the screens illustrated inFIG.29. The process illustrated inFIG.30starts when the screen illustrated inFIG.29(a)is displayed.

S41: The administrator 8 logs in to the information processing system10and inputs an operation of displaying a list of applications from the initial screen630to the terminal apparatus30. The operation acceptance unit33of the terminal apparatus30accepts the operation, and the first communication unit31sends a request for a list of applications to the information processing system10.

S42: The communication unit19of the information processing system10receives the request for a list of applications, and the terminal screen providing unit42acquires, from the user management unit44, an application associated with the user ID of the administrator 8. The terminal screen providing unit42creates screen information of an application list screen including an application available to the administrator 8 for each external service system40. The communication unit19transmits the screen information of the application list screen to the terminal apparatus30.

S43: The administrator 8 selects an application to be associated with a setting file. The operation acceptance unit33of the terminal apparatus30accepts the selection. After that, as described above, the terminal apparatus30displays the association screen640.

S44: The administrator 8 presses the file selection button644on the association screen640. The operation acceptance unit33accepts the pressing, and the first communication unit31transmits to the information processing system10a request for a list of files stored in the external service system40set on the association screen640.

S45: The communication unit19of the information processing system10receives the request for the list of files, and the application execution unit46designates the external service system40and causes the input/output processing unit48to make a request for the list of files. The input/output processing unit48sends a request for the list of files to the external service processing unit62corresponding to the external service system40. The external service processing unit62acquires, from the user management unit44, the authentication information of the administrator 8 for the external service system40associated with the application or set by the user on the association screen, and transmits a request for the list of files to the external service system40.

S46: The external service system40permits the login of the administrator 8 based on the authentication information of the administrator 8, and acquires a list of files associated with the administrator 8. The list of files is transmitted to the terminal apparatus30. In the process illustrated inFIG.30, the external service system40that stores the setting file and the external service system40that stores the file of the data acquired from the area are the same. However, both external service systems40may be different. In a case where both external service systems40are different, after selecting a setting file, the administrator 8 changes the external service system40on the association screen640.

S47: The display control unit32of the terminal apparatus30displays the list of files. The administrator 8 selects a setting file to be associated with the application. The operation acceptance unit33of the terminal apparatus30accepts the selection.

S48: The first communication unit31of the terminal apparatus30designates the identification information of the setting file, the identification information of the application, the external service system40, and the upload destination folder, and transmits a request for associating the setting file with the application to the information processing system10.

S49: The communication unit19of the information processing system10receives the request for associating the setting file with the application, and the application data management unit41registers the setting file and the application in the application data43in association with each other. That is, the application data management unit41registers the pieces of information in the “applicationId”, “applicationName”, “fileId”, “filename”, and “Upload destination folder” field in association with each other (see Table 3). The external service system40is registered in the application processing information53.

As described above, the application and the setting file are associated with each other. After that, the general user can output data of an area of the form to a file simply by pressing the icon from the device20.

WhileFIG.30illustrates a method in which the administrator 8 associates an application with area information on the association screen640, an application and area information may be associated with each other upon registration of the area information.

In this case, for example, the area registration screen610illustrated inFIG.24displays a field for entering an application name. The administrator 8 enters the application name of an application to be associated with a setting file. After the setting file is created, the information processing system10can associate the application having the entered application name with the setting file. That is, in response to the administrator 8 setting an application name and causing a form to be read, a setting file is generated, and the source file and the setting file are associated with each other.

The administrator 8 can set the extension width of the area on the association screen640illustrated inFIG.29(c).FIG.31illustrates a modification of the association screen640. The difference between the association screen640illustrated inFIG.29(c)and the association screen640-2illustrated inFIG.31will mainly be described.

The association screen640-2illustrated inFIG.31further includes a field645for setting the extension width of the area. The administrator 8 can set, in the field645for setting the extension width of the area, how much the area information (OCR area) is to be extended from the area identified with the marker pen. As a result, the area indicated by the area information in Table 4 is wider than the area colored with the marker pen. The OCR processing unit68can perform OCR processing on the characters in the area even when the area designated by the administrator 8 with the marker pen is narrower than the area to be marked or the user writes outside the area of the form.

However, without being limited to this example, the administrator 8 may set, on the association screen640-2, whether the area is limited to the inside of borders given by a marker (e.g., when the OCR area is surrounded by marker lines, a portion corresponding to the marker lines is excluded from the area) or whether the area includes a portion corresponding to the marker lines.

In addition, the color, the pen type, the shape, and the like to be used by the administrator 8 to designate an area may be accepted on the association screen640-2. For example, the administrator 8 can perform setting such that the area is surrounded by a square using a yellow marker, circled with a red pen, or filled in with a green marker.

Copying of Application

Forms have various formats, and areas to be filled in with data by users are located in different places depending on the form. In this embodiment, the administrator 8 can register area information in accordance with the format of each form. A method for registering area information for each form will be described in context of copying of an application by the administrator 8.

FIG.32illustrates a selection screen650for several applications. In one example, the applications include a specification application, a delivery statement application, and an OCR designation area application. The specification application is associated with area information of the specification in the application data43, and the delivery statement application is associated with area information of the statement of payment. A user who desires to acquire data from an area of the specification can activate the specification application and scan the specification to output data of the expected area. A user who desires to acquire data from an area of the delivery statement can activate the delivery statement application and scan the statement of payment to output data of the expected area.

The processes to be performed by the respective applications are described in the application processing information53. The process to be performed by the specification application and the process to be performed by the delivery statement application are executed by the same component although the storage locations of files, the upload destination folders, and the setting files are different. Accordingly, the administrator 8 copies the specification application or the delivery statement application, which allows the information processing system10to create an application corresponding to a desired form upon designation of an area, acquisition of an area, creation of a setting file, and association of the setting file with the copied application.

Copying of an application will be described with reference toFIG.33.FIG.33illustrates an example of screens operated by the administrator 8 to copy an application.FIG.33(a)illustrates the initial screen630displayed on the terminal apparatus30in response to the administrator 8 logging in to the information processing system10. The initial screen630displays a setting button651for the administrator 8 to perform various settings.

FIG.33(b)illustrates an application management screen652displayed after the setting button651is pressed. The application management screen652displays a group name653of a group of several applications. The illustrated screen transition is an example for description, and any of the illustrated screens may be omitted or a new screen may be added.

A list of applications in the group is displayed to the right of the group name653. The applications displayed in the list are applications that the administrator 8 is authorized to use. The applications include an application dedicated to the administrator 8, and an application to be used by other users. The administrator 8 presses a setting button654of an application from which a copy is created. In response to pressing of the setting button654, a menu is displayed. The menu includes a “copy” button655.

In response to the administrator 8 pressing the “copy” button655, a copy application setting screen (pop-up screen)660illustrated inFIG.33(c)is displayed. The copy application setting screen660includes an application name setting field661, an icon list field662, and a copy request button663. In the application name setting field661, the administrator 8 enters an application name for a new application created by copying. The application name may be any name, where the same application name is rejected or is followed by a number such as “(2)”. The application name may also be referred to as an icon name or a setting name instead of the application name.

The icon list field662displays a list of icons. The administrator 8 can select a desired icon. Pressing a reference button664may cause another icon to be displayed. In response to the administrator 8 pressing the copy request button663, the application ID, the application name, and the icon of the source application from which a copy is created are transmitted to the information processing system10. The information processing system10receives the application ID, the application name, and the icon, and the application management unit49registers a new application by using the application processing information53and the screen information54of the source application. An application ID that does not overlap with the application IDs in the application processing information53is used, and an application name entered by the administrator 8 is set as the application name. The same screen information as the screen information54of the source application is used, but the icon is selected by the administrator 8. The “application processing information”, “shared application”, and “external service system” items are the same as those of the source application. For example, if the user is able to identify an application using the name of the application or the like, not all applications may be assigned different icons.

In this embodiment, the setting of a new image processing area is added to copy an application. Alternatively, the administrator 8 may perform setting of associating addition of a new image processing area with identification information for which selection is accepted by a device, by, instead of copying an application, adding a setting or adding an icon.

FIG.34is a sequence chart illustrating an example procedure for the information processing system10to register a copied application.

S51: The administrator 8 inputs authentication information for the information processing system10to the terminal apparatus30. The operation acceptance unit33of the terminal apparatus30accepts the input of the authentication information, and the first communication unit31transmits the authentication information to the information processing system10. As a result, the administrator 8 logs in to the information processing system10.

S52: In response to the login, the terminal screen providing unit42of the information processing system10transmits screen information of an initial screen to the terminal apparatus30via the communication unit19. Then, in response to an operation by the administrator 8, the terminal screen providing unit42transmits screen information of the screens illustrated inFIG.33to the terminal apparatus30.

S53: The administrator 8 inputs the application name and the like to the terminal apparatus30, and the operation acceptance unit33accepts the input.

S54: The first communication unit31of the terminal apparatus30transmits the application ID, the application name, and the icon of the source application from which a copy is created to the information processing system10.

S55: The communication unit19of the information processing system10receives the application ID, the application name, and the icon, and the application management unit49registers a new application by using the application processing information53and the screen information54of the source application.

As a result, the administrator 8 can copy the application. The administrator 8 can associate a setting file with the new application generated by copying and acquire data from a predetermined area of a desired form.

The administrator 8 can create a plurality of new applications from one source application, and the information processing system10can accept selection of different setting files for the plurality of new applications. Copying of an application is an example. In this embodiment, any identification information may be assigned to the same process flow. In some embodiments, addition of an icon or addition of a setting may be implemented to assign identification information to the same process flow.

Acquisition of Data from Form using Association between Application and Setting File After the association between the application and the setting file is completed, the user causes the device20to read the form. As a result, the information processing system10can acquire data from the form.

Acquisition of data from a form will be described with reference toFIGS.35and36.FIG.35illustrates an example transition of screens displayed on the device20.

(1) As illustrated inFIG.35(a), the user causes the device20to display an application list screen. For example, a user who desires to reimburse expenses selects the icon of the expense claim form application. While the user knows which form corresponds to which application, it is desirable that application names displayed on icons690of applications coincide with the names of forms (documents). The application names may also be referred to as icon names or setting names. That is, the icons690may include the names of forms. Further, the icons690may include thumbnails of the forms. This may prevent the user from selecting an icon that does not correspond to a form. Alternatively, in response to the user pressing one of the icons690, the thumbnail of the corresponding form may be displayed. The user can check the pressed icon690by enlarging the thumbnail or displaying a portion of the thumbnail.

Each of the icons690may have an embedded link. The link includes, for example, a URL including the corresponding application ID or a URL associated with the corresponding application ID by the information processing system10. The device20connects to the URL upon pressing of the icon690to notify the information processing system10of the corresponding application ID.

In response to the user selecting a form that does not correspond to any of the icons690, the device20may provide a warning to the user. For example, the device20compares the image data of a read form with the image data associated with the icons690. If the image data of the read form does not match the image data associated with any of the icons690, the device20displays a message or the like to inquire of the user whether to continue the processing. The image data of the read form and the image data associated with the icons690may be compared by the information processing system10. In one example method for comparison, the device20or the information processing system10performs pattern matching of the image data of the read form against the image data associated with the icons690. In an alternative method, the device20or the information processing system10may compare an application ID printed on a barcode or the like on a form with the application IDs associated with the icons690.

Alternatively, the user may select an icon, and the device20may transmit a character string such as an application ID or a setting number to the information processing system10. That is, an application ID (identification information) may not necessarily be selected.

(2) When the device20is to scan an expense claim form, the user prepares the expense claim form (FIG.35(b)). The expense claim form may be prepared by the user by handwriting or may be printed from the device20. The expense claim form may not necessarily be printed by the user. Examples of such a document not printed by the user include an officially issued document and a document sent from another company.

(3) An application selected by the user displays a screen illustrated inFIG.35(c), based on the screen information54. The screen illustrated inFIG.35(c)is the same as the area registration screen610illustrated inFIG.24. The user sets a printed expense claim form on the contact glass and presses the start button613to cause the device20to read the expense claim form.

The image data of the form is transmitted to the information processing system10, and a file is output through a process described below.

FIG.36illustrates a file stored in the external service system40.FIG.36(a)illustrates an example of an upload destination folder in the external service system40. That is, the file output from the information processing system10is stored in the upload destination folder set by the administrator 8 on the association screen640illustrated inFIG.29. The file hierarchy “My Drive>expense claim form storage site” inFIG.36(a)is an example, and the administrator 8 can designate a folder for which the administrator 8 has the access right.

FIG.36(b)illustrates two files to be stored in the upload destination folder. Two files are generated in a single read operation.

A. A CSV file681in which data acquired from an area of the form is to be stored.

B. A PDF file682including image data of the form.

A different CSV file is created by the same application each time a read operation is performed. The file is not overwritten with the latest one because file names with serial numbers or dates are used. However, the user may desire to add pieces of data acquired by the same application to the same CSV file even if the pieces of data are read at different timings. This method will be described below. Thus, once accessing one CSV file, a person in charge such as a general affairs clerk can perform reimbursement or other processing on the data of the employees.

Case Where Device Does Not Scan Form

InFIG.35, the device20scans an expense claim form. The information processing system10may receive image data of the form from the terminal apparatus30or the like and acquire data from an area of the image data. In this case, the terminal apparatus30transmits the image data of the form to the information processing system10in the following way.

The user fills in items on an expense claim form displayed in a word processor operating on the terminal apparatus30. Accordingly, the items are represented by character code. The user connects the terminal apparatus30to the information processing system10and logs in to the information processing system10, and an application list is displayed on the terminal apparatus30. A screen similar to the screen illustrated inFIG.35(a)is displayed on the terminal apparatus30, and the user selects a desired application. The terminal screen providing unit42of the information processing system10transmits to the terminal apparatus30screen information that can accept the expense claim form. In response to the user selecting a PDF file or the like including the image data of the expense claim form on the screen, the terminal apparatus30can transmit the PDF file to the information processing system10.

As a result, the information processing system10can acquire data from the form without causing the user to print the form. In this case, the processing of the device20in the processes illustrated inFIGS.37A and37Bis performed by the terminal apparatus30.

Operation Procedure

FIGS.37A and37Bare sequence charts of an example procedure for the information processing system10to receive a PDF file including image data of a form from the device20and obtain acquired data from an area of the form.

S61: After printing the form, the user operates the device20to input their authentication information for the information processing system10. The operation acceptance unit23of the device20accepts the input of the authentication information, and the second communication unit21transmits the authentication information to the information processing system10. Accordingly, the user logs in to the information processing system10. In response to the login, the display control unit22of the device20displays a list of applications that the user is authorized to use (FIG.35(a)). For example, the user selects an application having an application name that coincides with the name of the printed form. The operation acceptance unit23accepts the selection.

S62: The second communication unit21of the device20designates an application ID and transmits an application screen acquisition request to the information processing system10.

S63: The communication unit19of the information processing system10receives the application screen acquisition request, and the device screen providing unit47acquires, from the application management unit49, the screen information54of the application identified by the application ID. Screen information of an application screen of the application is transmitted to the device20. Accordingly, the display control unit22of the device20displays the application screen (FIG.35(c)).

S64: The user sets the printed form on the contact glass and presses the start button613. The operation acceptance unit23of the device20accepts the pressing, and the image data generation unit24generates image data of the form and converts the image data into a PDF file. A plurality of forms may be used.

S65: The second communication unit21of the device20designates the application ID of the application and the PDF file and requests the information processing system10to execute the application.

S66: The communication unit19of the information processing system10receives the request to execute the application, and the application execution unit46starts executing the application. The application execution unit46designates an application ID and requests the input/output service unit16to execute the application. The input/output service unit16acquires the application processing information53via the application management unit49. As a result, the processing flow of the application is obtained. The input/output processing unit48further acquires the application data43associated with the application ID from the application data storage unit13. Accordingly, the input/output processing unit48can identify the setting file associated with the application.

The input/output processing unit48requests the external service processing unit62to acquire a setting file, based on the application data43and the application processing information53.

S67: The external service processing unit62connects to the URL in the “fileName” field registered in the application data43and acquires the setting file from the external service system40. If the application for which execution is requested is not a shared application, the external service processing unit62logs in to the external service system40using the authentication information of the user for the external service system40. When the application for which execution is requested is a shared application, the external service processing unit62logs in to the external service system40using the authentication information of the administrator 8 for the external service system40.

S68: Then, the input/output processing unit48requests the data storing unit66to acquire area information from the setting file, based on the application processing information53.

S69: The data storing unit66acquires all pieces of area information from predetermined rows and columns of the setting file and returns the acquired pieces of area information to the input/output processing unit48. The pieces of area information are registered on a process by process basis, and as many pieces of area information as the number of processes are acquired.

S70: Then, the input/output processing unit48designates the image data and the area information, based on the application processing information53, and requests the area identifying unit67to acquire (clip) an area from the image data.

S71: The area identifying unit67identifies and acquires a processing-target area designated by the area information from the image data, and returns a number of images of processing-target areas equal to the number of processes to the input/output processing unit48. The area identifying unit67may not necessarily acquire (clip) processing-target areas to be subjected to OCR processing and OMR processing, from the image data. In this case, the OCR processing unit68and the OMR processing unit71perform OCR processing and OMR processing on the processing-target areas of the image data identified by the area information, respectively.

In response to a plurality of processing-target areas to be subjected to OCR processing, steps S72and S73are repeatedly executed the number of times equal to the number of processing-target areas.

S72: The input/output processing unit48designates an image of a processing-target area, based on the application processing information53, and requests the OCR processing unit68to perform OCR processing on the processing-target area. At this time, the input/output processing unit48may designate the format and the OCR language acquired from the setting file in association with the processing-target area.

S73: The OCR processing unit68performs OCR processing on the processing-target area using a recognition engine supporting the format and the OCR language. The OCR processing unit68returns the data converted into character code data (OCR processing result) to the input/output processing unit48.

The recognition engine is, for example, an identifier that learns the correspondence between images of characters and character code sets using training data including pairs of the images and the character code sets and that outputs an OCR processing result (character code data) from an image. Simple training data, such as training data including only Japanese text, training data including only English (alphabetic) text, training data including only numerical values, or training data including only dates, is used to increase the identification accuracy of the identifier. In contrast, use of training data including a plurality of types such as Japanese text, English (alphabetic) text, numerical values, and dates tends to decrease the identification accuracy although various types of data can be recognized. Accordingly, the OCR processing unit68can use a recognition engine supporting the format and the OCR language to improve the recognition accuracy.

As used herein, machine learning is a technique that enables a computer to acquire human-like learning ability. Machine learning refers to a technology in which a computer autonomously generates an algorithm to be used for determination such as data identification from training data loaded in advance and applies the generated algorithm to new data to make a prediction. Any suitable learning method is applied for machine learning, for example, any one of supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, and deep learning, or a combination of two or more of those learning methods. Further, the recognition engine may not necessarily be generated by machine learning.

In response to a plurality of processing-target areas to be subjected to OMR processing, steps S74and S75are repeatedly executed the number of times equal to the number of processing-target areas.

S74: The input/output processing unit48designates an image of a processing-target area, based on the application processing information53, and requests the OMR processing unit71to perform OMR processing on the processing-target area.

S75: The OMR processing unit71performs OMR processing on the processing-target area. The OMR processing unit71returns acquired data (OMR processing result), which is converted into information indicating the presence or absence of a mark, to the input/output processing unit48.

In response to a plurality of processing-target areas to be subjected to trimming processing, steps S76and S77are repeatedly executed the number of times equal to the number of processing-target areas.

S76: The input/output processing unit48designates an image of a processing-target area, based on the application processing information53, and requests the trimming processing unit72to perform trimming processing on the processing-target area.

S77: The trimming processing unit72performs trimming processing on the processing-target area. The trimming processing unit72returns image data (trimming processing result) to the input/output processing unit48as acquired data obtained from the processing-target area.

S78: Then, the input/output processing unit48designates the acquired data, the CSV column name, the CSV position, and the file name (acquired from the setting file), which are acquired from each processing-target area, based on the application processing information53, and requests the data storing unit66to output a file including the acquired data.

S79: The data storing unit66writes the CSV column name to the beginning of the column, writes the acquired data to a cell of a CSV file designated by the CSV position, and returns the CSV file to the input/output processing unit48. The CSV file is assigned a designated file name. The acquired data may not necessarily be output to a CSV file and may be output to a file in tabular format, a text file, or a file of any other application.

S80: Then, the input/output processing unit48designates the OCR processing result (CSV file), the format, and the OCR language, based on the application processing information53, and requests the OCR correction unit69to correct the OCR processing result. The format is acquired from the setting file presented in Table 4 and is set by the administrator 8.

FIG.38illustrates correction of the OCR processing result. When the format is set in the setting file, the OCR correction unit69can correct the OCR processing result. As illustrated inFIG.38, rules for designated character strings and converted characters are defined in the OCR correction unit69. A conversion rule2201illustrated inFIG.38is a rule for the numeric format. For example, I, B, Z, which are not numeric, are corrected to 1, 8, and 2, respectively. A conversion rule2202illustrated inFIG.38is a rule for the alphabetic format. For example, 1, 2, and 8, which are not alphabetic, are corrected to I, Z, and B, respectively.

S81: Referring back toFIG.37B, the OCR correction unit69returns the corrected OCR processing result (CSV file) to the input/output processing unit48.

Steps S82to S85are executed the number of times equal to the number of CSV files. The number of CSV files to be created in a single read operation is one, regardless of the number of forms, or the same as the number of pages of the form. The number of CSV files to be created in a single read operation is set in the application.

S82: Then, the input/output processing unit48requests the external service processing unit62to upload the OCR processing result (CSV file), based on the application processing information53. The upload destination folder is set in the application data43.

S83: The external service processing unit62stores the CSV file in the upload destination folder in the external service system40.

S84: If the metadata can be set, step S84is executed. When the external service system40supports setting of metadata (items set by the user on an external service system, such as a character string and a date), the user can set the metadata to be associated with the CSV file to be uploaded. The user can input metadata from a screen displayed on the device20. The input/output processing unit48requests the external service processing unit62to upload the metadata. The upload destination folder is the same as that of the CSV file.

S85: The external service processing unit62stores the metadata in the upload destination folder in the external service system40.

S86: In response to a plurality of pieces of form data being read, some users may desire to store the plurality of pieces of form data in individual files. In this case, the input/output processing unit48sends a request together with the image data of the plurality of pages to the PDF processing unit64to separate the pages, based on the application processing information53. This processing is optional. When this processing is omitted, the input/output processing unit48may store the plurality of pieces of data as a single file.

S87: The PDF processing unit64returns the PDF file decomposed into pages to the input/output processing unit48.

S88: The input/output processing unit48designates a PDF file, based on the application processing information53, and requests the OCR processing unit68to perform OCR processing.

S89: The OCR processing unit68performs OCR processing on the PDF file to generate character code data, and sets the character code data in the PDF file as transparent text.

Steps S90to S93are executed the number of times equal to the number of PDF files.

S90: The input/output processing unit48requests the external service processing unit62to upload the PDF file with text, based on the application processing information53. The upload destination folder is set in the application data43.

S91: The external service processing unit62stores the PDF file with text in the upload destination folder in the external service system40.

S92: When the external service system40supports the metadata described above (items set by the user on an external service system, such as a character string and a date), the user can input the metadata from a screen displayed on the device20. The input/output processing unit48requests the external service processing unit62to upload the metadata. The upload destination folder is the same as the PDF file.

S93: The external service processing unit62stores the metadata in the upload destination folder in the external service system40.

As described above, the CSV file and the PDF file, which is different from the CSV file, are registered in the upload destination folder in the external service system40.

InFIGS.37A and37B, the image data to be processed is a PDF file. Image data in any format such as JPEG, PNG, TIFF, or BMP may be to be processed.

In this embodiment, the device20is assumed to collectively scan and process a large volume of forms and the like. However, a form having the same format may be used by a user once, or the area to be processed may be changed each time a form having the same format is used. In this case, the registration of area information in a setting file is time-consuming to the administrator 8. Accordingly, the task of registering area information may be omitted, and areas of a form filled in by the user may be directly retouched and read by the device20. The information processing system10performs a process on each area in accordance with the correspondence between colors and processes registered in the setting file.

Display of CSV File and PDF File Registered in External Service System

A person in charge of a company operates the terminal apparatus30and connects to the external service system40to calculate expenses or the like incurred by employees. The person in charge, who is notified of the upload destination folder, can acquire the CSV file and the PDF file using the terminal apparatus30.

FIG.39illustrates an example of a file list screen670of the external service system40, which is displayed on the terminal apparatus30. As illustrated inFIG.39(a), a PDF file671and a CSV file672are stored in the upload destination folder.FIG.39(b)illustrates the content of the CSV file672displayed on the terminal apparatus30. That is,FIG.39(b)illustrates data output from the form. InFIG.39(b), data acquired from two areas is stored. Each column specifies a CSV column name set in the setting file, which allows the user to determine the data in the column.

As illustrated inFIG.40, it is preferable that the device20also displays the processing result.FIG.40illustrates an example of an output data display screen691displayed on the device20. The output data display screen691displays an application name692of an application that has been executed, and a list693of files output from the application. The application name692and the list693are transmitted from the information processing system10to the device20. The CSV file includes the content of the CSV file, and the user clicks the CSV file to cause the device20to display data694acquired from the areas of the form and stored in the CSV file.

The user checks the file name and the data to determine whether the form has been processed correctly.

Case where Multiple Forms Are Read

A case where a plurality of forms are read in a single read operation will be described with reference toFIG.41.FIG.41illustrates a plurality of forms and an output file. As illustrated inFIG.41(a), a plurality of forms are read in a single read operation in response to execution of the application. A single read operation refers to one click of the start button613. The device20sequentially reads the plurality of forms, and the information processing system10acquires a plurality of pieces of data from the same area in the image data of the respective forms.

FIG.41(b)illustrates a CSV file obtained in response to a plurality of forms being read in a single read operation. The first row specifies data acquired from the first form, and the second row specifies data acquired from the second form. In this manner, the plurality of pieces of data of the plurality of forms are integrated into a single CSV file, which allows the person in charge to easily tabulate or graph the results.

When a plurality of forms are read in a single read operation, the data storing unit66increments the CSV position (start row) in the setting file one by one. That is, the data storing unit66may shift a row down for each form and write the data of the form in the row.

When a plurality of forms are read in a single read operation, the information processing system10may output a number of CSV files equal to the number of forms.

Alternatively, when a plurality of forms are read in a plurality of read operations (when a plurality of forms are read at different timings), the information processing system10may output data to the same CSV file. The CSV files may be merged by the external service system40or the information processing system10.

In a case where CSV files are merged by the external service system40, the external service processing unit62requests the external service system40to merge the CSV files. Since CSV files output from the same application have file names including the same character string (e.g., “expense claim form”), the external service processing unit62can designate a file name and request the external service system40to merge the CSV files. The external service system40adds data in column numbers of a requested CSV file to the corresponding column numbers of a stored CSV file.

In a case where CSV files are merged by the information processing system10, the external service processing unit62designates a file name and downloads a CSV file from the external service system40. The external service processing unit62adds data in column numbers of a newly created CSV file to the corresponding column numbers of the acquired CSV file. The external service processing unit62uploads a CSV file obtained by merging the CSV files to the external service system40.

As described above, the information processing system10can acquire the same area from a plurality of pieces of received image data, the OCR processing unit68can perform OCR processing on the same area in the plurality of pieces of image data, and the data storing unit66can output a plurality of pieces of data generated from the same area in the plurality of pieces of image data to one file.

Advantages

As described above, the data output system100according to this embodiment enables the administrator 8 to associate a mark and a process with each other on a registration sheet using a marker pen. The device20reads a form with a marked area, and the information processing system10stores a process and area information for each form. Then, the information processing system10acquires an area from the image data of the form transmitted from the device20, and performs a process set for the area. As a result, it is possible to easily create a file in which a specific area is extracted from a plurality of forms of the same type.

Second Embodiment

This embodiment presents an information processing system10for recognizing a different recognition target using OCR in accordance with the color of an area.

An overview of processes or operation performed by the data output system100will be described with reference toFIGS.42,43A, and43B. In the description ofFIGS.42,43A, and43B, differences fromFIGS.1,2A, and2Bwill mainly be described.

In the processes illustrated inFIG.42, the process (2) is different from that illustrated inFIG.1.

(2) An administrator 8 of the data output system100prints a registration sheet registered in an information processing system10using a device20, and colors the registration sheet with a color corresponding to a process using a marker pen. For example, the administrator 8 can associate processes with certain colors such that “OCR of Kanji characters” is associated with green, “OCR of dates” is associated with blue, and “OCR of alphanumeric characters” is associated with red. The device20scans a registration sheet to generate image data (first image data), and transmits the image data to the information processing system10.

Next, a description will be given with reference toFIGS.43A and43B.FIG.43Aillustrates setting of area information in a form, andFIG.43Billustrates acquisition of data from the form using the area information. The processes illustrated inFIGS.43A and43Bare similar to the processes illustrated inFIGS.2A and2B, and differences therebetween will be described.

(4) The administrator 8 uses a form 7, which is usually used by users in the organization, and designates, with a marker pen, which of the processes (e.g., OCR of Kanji characters, OCR of dates, and OCR of alphanumeric characters) is to be performed on each area of the form 7.

The subsequent processes (5) to (11) may be similar to those illustrated inFIGS.2A and2B.

The data output system100according to this embodiment provides the advantages of the first embodiment and enables the administrator 8 to register more detailed content of a process and a mark (color). It is known that the recognition rate of OCR greatly varies depending on whether to limit the recognition targets. For example, when the OCR processing unit68performs OCR on the Japanese postal code mark “”, the mark may be incorrectly recognized as the character “T” unless it is understood in advance that the recognition target is the postal code. In this embodiment, a target to be recognized by OCR is set for each area, and OCR recognition accuracy can be improved.

While it is possible to adopt a method in which the administrator 8 uses a web site to set what is to be filled in an area of a form, this embodiment adopts a simple method in which the administrator 8 marks a registration sheet to set a target to be recognized by OCR for each area.

Supplementary Information Related to Correspondence between Color and Process The series of processes described with reference toFIGS.42,43A, and43Bwill be divided into tasks 1 to 3 to provide supplementary information.FIG.44illustrates registration (task 1) of a correspondence between a color and a process using a registration sheet.

FIG.44(a)illustrates the registration sheet before coloring is applied.

FIG.44(b)illustrates the registration sheet after coloring is applied. The registration sheet illustrated inFIG.44(b)is scanned by the device20.

FIG.44(c)illustrates a portion of a setting file in which colors and processes are associated with each other. Two methods are conceivable for associating colors with processes.

In a first method, information indicating whether to associate the coordinates (in the registration sheet) at which a color is acquired with the process obtained from the coordinates is stored in the information processing system10in advance. Example parameters prepared in advance are as follows: Color extracted at coordinates (X coordinates: 100, 200; Y coordinates: 1, 50) for “OCR of Kanji characters”; Color extracted at coordinates (X coordinates: 100, 200; Y coordinates: 51, 100) for “OCR of dates”; and Color extracted at coordinates (X coordinates: 100, 200; Y coordinates: 101, 150) for “OCR of alphanumeric characters”. In a second method, the administrator 8 associates colors with processes on the management site.FIG.45illustrates an example of a management site150for associating a color with a process. The management site150is provided as a web page or a web application from the information processing system10to the terminal apparatus30. The administrator 8 can register colors for processes using radio buttons171. In the illustrated example, the user uses three marker pen colors.

In either method, as illustrated inFIG.44(c), a numerical value identifying the marker color (green) and “OCR of Kanji characters” are associated with each other. A numerical value identifying the marker color (blue) and “OCR of dates” are associated with each other. A numerical value identifying the marker color (blue) and “OCR of alphanumeric characters” are associated with each other.

FIG.46illustrates a correspondence (task 2) between area information and a process using the original of a form.FIG.46(a)illustrates the original of the form.

FIG.46(b)illustrates the form after coloring is applied. The form illustrated inFIG.46(b)is scanned by the device20.FIG.46(c)illustrates a portion of a setting file in which area information and processes are associated with each other.

As illustrated inFIG.46(d), colors and processes are associated with each other in the setting file. The information processing system10detects a color of the registration sheet (registered in the setting file) from the form, and associates the coordinates of the area where the color is detected with the process in the setting file as area information.

In this embodiment, accordingly, the administrator 8 can change a color used for an area of the form to change the content of OCR processing to be performed on the area.

Ex1: colored in blue→OCR of Kanji characters, colored in red→OCR of dates

Ex2: colored in blue→OCR of postal code marks, colored in red→OCR of URLs

FIG.47illustrates a process (task 3) to be performed on an area in a form using the form.

FIG.47(a)illustrates the original with items filled in by the user.

FIG.47(b)illustrates processes to be performed on respective areas of the form. Since a process of performing OCR of Kanji characters is registered in a name area172of the form as OCR1, an entered name is identified as Kanji characters (including katakana and hiragana characters) and is subjected to OCR. Since a process of performing OCR of dates is registered in an application date area173of the form as OCR2, an entered application date is identified as the date and time and is subjected to OCR. Since a process of performing OCR of alphanumeric characters is registered in an employee number area174as OCR3, an entered employee number is identified as alphanumeric characters and is subjected to OCR.

Functions

In this embodiment, the hardware configuration diagrams illustrated inFIGS.8and9and the functional configuration diagram illustrated inFIG.10according to the first embodiment described above are applied.

FIG.48illustrates the functional configuration of the OCR processing unit68according to this embodiment. The OCR processing unit68includes a first OCR processing unit68a, a second OCR processing unit68b, and a third OCR processing unit68c.

The first OCR processing unit68aperforms character recognition of a Kanji character. The second OCR processing unit68bperforms character recognition of a date. The third OCR processing unit68cperforms character recognition of an alphanumeric character. The first OCR processing unit68a, the second OCR processing unit68b, and the third OCR processing unit68care identification devices that have learned the correspondence between images of Kanji characters and character codes, the correspondence between images of dates and character codes, and the correspondence between images of alphanumeric characters and character codes, respectively, by using machine learning.

Machine learning is a technique that enables a computer to acquire human-like learning ability. Machine learning refers to a technology in which a computer autonomously generates an algorithm to be used for determination such as data identification from training data loaded in advance and applies the generated algorithm to new data to make a prediction. Any suitable learning method is applied for machine learning, for example, any one of supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, and deep learning, or a combination of two or more of those learning methods.

As presented in Table 5, in this embodiment, a color of an area and an OCR language are associated with each other in a setting file.

Table 5 presents an example of the setting file 9 stored in the external service system40according to this embodiment. In the description of Table 5, differences from Table 4 will mainly be described. In the setting file presented in Table 5, the “process” item specifies OCR1, OCR2, and OCR3. The “OCR language” item specifies recognition targets corresponding to colors of areas. In Table 5, for example, “Kanji character” is set for green, “date” is set for blue, and “alphanumeric character” is set for red.

The first OCR processing unit68aperforms character recognition of a Kanji character in an area colored in green. The second OCR processing unit68bperforms character recognition of a date in an area colored in blue. The third OCR processing unit68cperforms character recognition of an alphanumeric character in an area colored in red.

In this embodiment, accordingly, the recognition targets are limited in accordance with Kanji characters, dates, and alphanumeric characters in the corresponding areas, and recognition accuracy can be improved.

The type (recognition target) of OCR is not limited to those described above, and examples thereof include a postal code, an email address, a single-byte alphanumeric character, a number (such as a telephone number or a postal code in a format without a hyphen between numbers), a number (such as a telephone number or a postal code in a format with a hyphen between numbers), and a URL. In this embodiment, Kanji characters include character strings in hiragana and katakana. Text in various languages (such as English, French, German, Chinese, and Arabic) may be recognition targets.

Application Processing Information

Next, application process information will be described in detail. The application processing information illustrated inFIG.11is also applied to this embodiment.

FIGS.49A to49Cschematically illustrate series of processes performed by applications. In the description ofFIGS.49A to49C, differences fromFIGS.12A to12Cwill mainly be described.FIG.49Cillustrates a series of processes of an application (e.g., an expense claim form application) for outputting data from an area of a form. In this embodiment, OCR of Kanji characters, dates and times, or alphanumeric characters is performed in accordance with the OCR language (recognition target) set in the setting file.

Registration of Correspondence between Color and Process in Setting File

Next, setting of a correspondence between a color and a process by the administrator 8 and creation of a setting file will be described with reference toFIG.50.

FIG.50illustrates a method for setting a correspondence between a color and a process using a registration sheet. In the description ofFIG.50, differences fromFIG.13will mainly be described.FIG.50(a)illustrates a registration sheet before the administrator 8 colors the registration sheet with marker pens. The original of the registration sheet is registered in the external service system40.

The first row of the registration sheet specifies items of the respective columns (marker color in the first column, and process in the second column). This embodiment includes three “processes” (OCR of Kanji characters, OCR of dates, and OCR of alphanumeric characters), and the following processes are set in the registration sheet in advance: OCR of Kanji characters in the second row; OCR of dates in the third row; and OCR of alphanumeric characters in the fourth row. In the original of the registration sheet, cells130designating marker colors are blank.

FIG.50(b)is a view in which the administrator 8 colors the cells130with marker pens for the respective processes. For example, the administrator 8 uses green, blue, and red marker pens among the owned marker pens. The administrator 8 assigns green to OCR of Kanji characters, blue to OCR of dates, and red to OCR of alphanumeric characters. The original of the registration sheet includes predetermined spacings between rows to avoid mixture of color inks due to spreading of the color inks.

The administrator 8 colors each of the cells130of the registration sheet with a marker pen among the owned marker pens in accordance with the assignment of the desired marker color to the process, which is determined by the administrator 8. The device20scans the registration sheet to register the correspondence between the marker colors and the processes in the information processing system10.

Screen Example on Device

An example screen on the device20may be similar to that illustrated inFIG.14. A sequence chart illustrating an example procedure for the administrator 8 to register a correspondence between a color and a process in a setting file using a registration sheet may be similar to that inFIG.15. However, the processing of step S12is different.

S12: The color detection unit73detects color information from the cells130determined in advance in the image data.

FIGS.51A and51Bschematically illustrate detection of color information. In the description ofFIGS.51A and51B, differences fromFIGS.16A and16Bwill mainly be described. As illustrated inFIG.51A, the process of OCR of Kanji characters is determined by the fill color of a rectangular cell defined by two position coordinate points A and B (an example of position information) indicating diagonal corners. The process of OCR of dates is determined by the fill color of a rectangular cell defined by position coordinate points C and D. The process of OCR of alphanumeric characters is determined by the fill color of a rectangular cell defined by position coordinate points E and F.

As illustrated inFIG.51B, the barcode101or the application data management unit41stores the processes and the sets of position coordinate points of the respective cells130to be colored in association with each other. The color detection unit73acquires the H value of the HSV color system from each of the cells130, based on the position coordinate points illustrated inFIG.51B. Upon detection of the color (H=(0 to 60°, 301 to 360°)) corresponding to red from the cell130associated with OCR of alphanumeric characters, the color detection unit73registers H=(0 to 60° , 301 to 360°) of the HSV color system in the setting file in association with OCR of alphanumeric characters. Upon detection of the color (H=(61 to 180°)) corresponding to green from the cell130associated with OCR of Kanji characters, the color detection unit73registers H=(61 to 180°) in the setting file in association with OCR of Kanji characters. Upon detection of the color (H=(181 to 300°)) corresponding to blue from the cell130associated with OCR of dates, the color detection unit73registers H=(181 to 300°) in the setting file in association with OCR of dates.

The colors and the processes may be associated with each other by the administrator using the management site150illustrated inFIG.45.

Designation of File Name using Process

Also in this embodiment, the administrator 8 may set the file name of the CSV file by coloring the registration sheet.

FIG.52illustrates an example of use of a registration sheet in which a process is applied to the setting of a file name. In the description ofFIG.52, differences fromFIG.17will mainly be described. As inFIG.17, the processes are limited to OCR. In the registration sheet illustrated inFIG.52(a), OCR1 and OCR2 are set as processes, and the order of use for the file names, that is, “File name 1” and “File name 2”, is set.

As illustrated inFIG.52(b), the administrator 8 uses marker pens to designate the color (e.g., green) of the marker pen to be used for File name 1 and the color (e.g., blue) of the marker pen to be used for File name 2.

As illustrated inFIG.52(c), a numerical value indicating a color and a process (to be used for a file name) are associated with each other in the setting file. Two methods similar to those inFIG.17for associating colors with processes may be used.

FIG.53illustrates an example of a management site152for associating a color with a process. The administrator 8 can use radio buttons153to register colors to be used to designate file names. In the illustrated example, the user uses three marker pen colors.

As illustrated inFIG.52(c), a numerical value identifying the marker color (green) and File name 1 are associated with each other. A numerical value identifying the marker color (blue) and File name 2 are associated with each other. File names 1 and 2 indicate the order of use for the file names.

FIGS.54A and54Billustrate an example of a form in which the administrator 8 designates two areas using marker pens.FIG.54Aillustrates the original of the form, andFIG.54Billustrates the form colored with marker pens. The device20scans the form illustrated inFIG.54B, and the information processing system10identifies the colored areas in the form and registers area information in the setting file.

InFIG.54B, the name area172is designated by a green marker pen, and the application date area173is designated by a blue marker pen. For example, when the name is “Ricoh Taro” and the application date is “2020/10/10”, the file name of the CSV file obtained when the form illustrated inFIG.54Bis scanned by the device20is “Ricoh Taro20201010.csv”. Three or more marker pen colors may be used to specify file names.

The order of areas to be used for file names (file 1, file 2) may not necessarily be registered. In this case, the data storing unit66sets the file names in the order of registration of the colors in the setting file. As a result, colors of marker pens are not changed from cell to cell of the registration sheet.

Modification of Registration Sheet

As illustrated inFIG.55, a CSV position may be registered in advance in the registration sheet. In the description ofFIG.55, differences fromFIG.20will mainly be described.FIG.55(a)illustrates the registration sheet before coloring is applied. The registration sheet includes columns of a CSV column name154and a CSV position155, and the administrator 8 does not manually set a CSV position in the setting file.

FIG.55(b)illustrates the registration sheet after coloring is applied. The registration sheet illustrated inFIG.55(b)is scanned by the device20.FIG.55(c)illustrates a portion of a setting file in which colors, processes, and area information are associated with each other.

Registration of Area Information

Next, registration of the area information will be described with reference toFIGS.56A to56Cand the like.FIGS.56A to56Cillustrate a method for marking areas in a form.FIG.56Aillustrates the form (original) before areas are designated using marker pens. The administrator 8 designates, on the original of the form, an area on which a process registered in the registration sheet is to be executed, using a marker pen of a registered color. For example, the administrator 8 is to set the name area172as an area to be subjected to OCR of Kanji characters, the application date area173as an area to be subjected to OCR of dates, and the employee number area174as an area to be subjected to OCR of alphanumeric characters.

FIG.56Billustrates a form sheet on which areas are colored with colors of marker pens. The name area172is colored with a green marker pen. The application date area173is colored with a blue marker pen. The employee number area174is colored with a red marker pen. The device20scans the form sheet to register the relationship between the areas of the form and the processes in the setting file.

According to the setting file presented in Table 5, the first OCR processing unit68aperforms OCR processing on the name area172such that Kanji characters are to be recognized to generate acquired data. The second OCR processing unit68bperforms OCR processing on the application date area173such that a date and time are to be recognized to generate acquired data. The third OCR processing unit68cperforms OCR processing on the employee number area174such that alphanumeric characters are to be recognized to generate acquired data.

FIG.56Cillustrates an example of a form with items filled in by the user. From the colored areas illustrated inFIG.56B, “Ricoh Taro”, “2020/10/10”, and “50412” are acquired data.

Area Registration Screen

FIG.57illustrates an example of an area registration screen610displayed on the device20upon registration of area information. In the description ofFIG.57, differences fromFIG.24will mainly be described. InFIG.57, areas marked on the form are different from those inFIG.24.

A sequence chart illustrating an example procedure for the information processing system10to acquire area information from a form with an area marked by the administrator 8 and create a setting file may be similar to that inFIG.25. The examples illustrated inFIGS.26A to26C and27according to the first embodiment are also applied.

FIG.58illustrates a correspondence between a setting file and a form. In the example illustrated inFIG.58, the data acquired from the area information with ID=1 corresponds to the CSV column name “name”. The item “name” is used first in the file name. The data acquired from the area information with ID=2 corresponds to the CSV column name “application date”. Since the CSV position indicating the application date is set to column 3 and row 2, the data acquired from the area information with ID=2 is written to the third column and second row of the file. The data acquired from the area information with ID=3 corresponds to the CSV column name “employee number”. Since the CSV position indicating the employee number is set to column 4 and row 2, the data acquired from the area information with ID=3 is written to the fourth column and second row of the file.

The examples illustrated inFIGS.29to36according to the first embodiment are also applied.

Operation Procedure

FIGS.59A and59Bare sequence charts of an example procedure for the information processing system10to receive a PDF file including image data of a form from the device20and obtain acquired data from an area of the form. In the description ofFIGS.59A and59B, differences fromFIGS.37A and37Bwill mainly be described.

The processing of steps S61to S71may be similar to that inFIG.37A.

In response to a plurality of processing-target areas to be subjected to OCR processing, steps S72and S73are repeatedly executed the number of times equal to the number of processing-target areas.

S72: The input/output processing unit48designates an image of a processing-target area, based on the application processing information53, and requests the OCR processing unit68to perform OCR processing on the processing-target area. In this embodiment, the input/output processing unit48may designate the format and the OCR language (Kanji characters, dates, and alphanumeric characters) acquired from the setting file in association with the processing-target area.

S73: The OCR processing unit68performs OCR processing on the processing-target area using a recognition engine supporting the format and the OCR language. That is, the first OCR processing unit68arecognizes a Kanji character, the second OCR processing unit68b recognizes a date, and the third OCR processing unit68crecognizes an alphanumeric character. The OCR processing unit68returns the data converted into character code data (OCR processing result) to the input/output processing unit48.

S78, S79: Then, the input/output processing unit48designates the acquired data, the CSV column name, the CSV position, and the file name (acquired from the setting file), which are acquired from each processing-target area, based on the application processing information53, and requests the data storing unit66to output a file including the acquired data.

The processing of steps S80to S93may be similar to that inFIG.37B.

As a result, the CSV file and the PDF file registered in the external service system40can be displayed. The method for displaying the CSV file and the PDF file may be similar to that in the first embodiment.

As described above, the data output system100according to this embodiment provides the advantages of the first embodiment and enables the administrator 8 to register more detailed content of a process and a mark (color). A target to be recognized by OCR is set for each area, and OCR recognition accuracy can be improved. Further, this embodiment adopts a simple method in which the administrator 8 marks a registration sheet to set a target to be recognized by OCR for each area.

For example, in the embodiments described above, the information processing system10outputs data acquired from an area of a form to a file. In one or more embodiments, data acquired from an area of a form may be transmitted by email instead of being formed into a file or may be displayed on the device20.

In the example configurations illustrated inFIG.10and the like, the terminal apparatus30, the device20, and the information processing system10are each divided according to main functions to facilitate understanding of the processes performed by the terminal apparatus30, the device20, and the information processing system10. No limitation is intended by how the functions are divided by process or by the name of the functions. The processing of the terminal apparatus30, the device20, and the information processing system10may be divided into more units of processing in accordance with the content of the processing. Further, the division may be made such that each unit of processing includes more processing.

The apparatuses or devices described in one or more embodiments are just one example of plural computing environments that implement the one or more embodiments disclosed herein. In some embodiments, the information processing system10includes multiple computing devices, such as a server cluster. The plural computing devices are configured to communicate with one another through any type of communication link including a network, shared memory, etc., and perform the processes disclosed herein.

Further, the information processing system10can be configured to share the processing steps disclosed in the embodiments described above, for example, the processing steps illustrated inFIGS.37A and37B, in various combinations. For example, a process executed by a predetermined unit may be executed by a plurality of information processing apparatuses included in the information processing system10. The components of the information processing system10may be integrated into one server apparatus or divided into a plurality of apparatuses.

Each of the functions of the described embodiments may be implemented by one or more processing circuits or circuitry. Processing circuitry includes a programmed processor, as a processor includes circuitry. A processing circuit also includes devices such as an application specific integrated circuit (ASIC), a digital signal processor (DSP), a field programmable gate array (FPGA), and conventional circuit components arranged to perform the recited functions.

The present invention can be implemented in any convenient form, for example using dedicated hardware, or a mixture of dedicated hardware and software. The present invention may be implemented as computer software implemented by one or more networked processing apparatuses. The processing apparatuses include any suitably programmed apparatuses such as a general purpose computer, a personal digital assistant, a Wireless Application Protocol (WAP) or third-generation (3G)-compliant mobile telephone, and so on.

Since the present invention can be implemented as software, each and every aspect of the present invention thus encompasses computer software implementable on a programmable device. The computer software can be provided to the programmable device using any conventional carrier medium (carrier means). The carrier medium includes a transient carrier medium such as an electrical, optical, microwave, acoustic or radio frequency signal carrying the computer code. An example of such a transient medium is a Transmission Control Protocol/Internet Protocol (TCP/IP) signal carrying computer code over an IP network, such as the Internet. The carrier medium also includes a storage medium for storing processor readable code such as a floppy disk, a hard disk, a compact disc read-only memory (CD-ROM), a magnetic tape device, or a solid state memory device.

This patent application is based on and claims priority to Japanese Patent Application Nos. 2021-040561, filed on Mar. 12, 2021, and 2021-090252, filed on May 28, 2021, in the Japan Patent Office, the entire disclosure of which is hereby incorporated by reference herein.

REFERENCE SIGNS LIST