INFORMATION PROCESSING SYSTEM, INFORMATION PROCESSING METHOD, AND NON-TRANSITORY COMPUTER READABLE MEDIUM

An information processing system includes a processor configured to, when an object in a screen is to be operated with a target in a contactless manner, accept an operation on the object that is in a selected state in response to the target being located in a second region closer to the screen than a first region among multiple regions separated according to a distance from the screen, the first region being assigned an operation for selecting the object in the screen, the second region being assigned an operation on the object.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is based on and claims priority under 35 USC 119 from Japanese Patent Application No. 2023-052423 filed Mar. 28, 2023.

BACKGROUND

(i) Technical Field

The present disclosure relates to an information processing system, an information processing method, and a non-transitory computer readable medium.

(ii) Related Art

Japanese Unexamined Patent Application Publication No. 2015-148960 discloses an information processing apparatus including position determination means for determining a position on a display surface of a display unit that displays information, the position on the display surface corresponding to the position of an operating object that is not in contact with the display surface and that is detected in a mid-air area close to the display surface; operation identification means for identifying an input operation based on a predetermined motion of the operating object detected in the mid-air area, and execution means for executing a predetermined process in accordance with the position determined by the position determination means and the input operation identified by the operation identification means.

SUMMARY

Aspects of non-limiting embodiments of the present disclosure relate to prevention or reduction of the occurrence of erroneous operations on an object in a screen to be operated with a target in a contactless manner, compared to when an operation on an object that is not in a selected state is accepted.

According to an aspect of the present disclosure, there is provided an information processing system including a processor configured to, when an object in a screen is to be operated with a target in a contactless manner, accept an operation on the object that is in a selected state in response to the target being located in a second region closer to the screen than a first region among a plurality of regions separated according to a distance from the screen, the first region being assigned an operation for selecting the object in the screen, the second region being assigned an operation on the object.

DETAILED DESCRIPTION

The following describes an exemplary embodiment of the present disclosure in detail with reference to the drawings. Components and processes that have substantially the same operations and functions are assigned the same reference symbols throughout the drawings, and redundant descriptions thereof may be omitted. The drawings are merely presented in schematic form to allow a full understanding of the present disclosure. Therefore, the present disclosure is not limited to only the illustrated examples. In the present exemplary embodiment, descriptions of configurations that are not directly related to the present disclosure or are well known may be omitted.

FIG.1is a diagram illustrating an example configuration of an information processing system1including an information processing apparatus having a contactless user interface through which a user performs an operation in a contactless manner.

The information processing apparatus in the information processing system1may be applied to any field as long as the information processing apparatus has a contactless user interface. Examples of the information processing apparatus include an image processing apparatus, an automatic teller machine (ATM), a vending machine, and an automatic ticket dispenser. The information processing apparatus may be for personal use only or usable by an unspecified number of users.

An image processing apparatus10installed in a workplace as an example of the information processing apparatus will be described hereinafter with reference toFIGS.1and2.

FIG.2is a perspective view of a substantial part of the image processing apparatus10according to the present exemplary embodiment.

As described below, the image processing apparatus10is configured to execute functions related to images in accordance with instructions from users. The image processing apparatus10is connected to, for example, a plurality of terminals4to be used by individual users via a communication line2.

Each user transmits image data generated by a corresponding one of the terminals4to the image processing apparatus10through the communication line2to cause the image processing apparatus10to execute desired image processing. Alternatively, a user may bring a portable storage medium such as a Universal Serial Bus (USB) memory or a memory card storing image data to the image processing apparatus10and connect the portable storage medium to the image processing apparatus10to cause the image processing apparatus10to execute desired image processing. Alternatively, a user may bring a document11having at least one of text or an image to the image processing apparatus10and make the image processing apparatus10read the document11to cause the image processing apparatus10to execute desired image processing.

The communication line2may be of any type that provides a connection between the image processing apparatus10and the terminals4, such as a wired connection, a wireless connection, or a combination of wired and wireless connections. In addition, any number of terminals4may be connected to the image processing apparatus10. For example, none of the terminals4may be connected to the image processing apparatus10.

The terminals4are information devices configured to be used by users. The terminals4may be any type of information device having a data storage function and a data communication function. The terminals4include, for example, computers intended to be used at fixed positions, and mobile terminals intended to be transported and used, such as smartphones and wearable devices.

As illustrated inFIG.2, the image processing apparatus10has, for example, a scan function for reading an image on a recording medium such as paper as image data, a print function for forming an image represented by image data on a recording medium, and a copy function for forming the same image as an image formed on a recording medium onto another recording medium. The copy function, the print function, and the scan function are examples of image processing to be performed by the image processing apparatus10.

The image processing apparatus10illustrated inFIG.2includes, for example, a document reading device12in an upper portion thereof, and an image forming device14below the document reading device12.

The document reading device12includes an optical reading device (not illustrated) and a document transport device18. The document transport device18is disposed in a document cover16. The document cover16is provided with a document table16A, on which documents11are placed. The document transport device18sequentially feeds each of the documents11on the document table16A and transports the document11onto a transported-document scanning glass (not illustrated). The document reading device12reads the content of the document11transported onto the transported-document scanning glass as image data using the optical reading device. Thereafter, the document transport device18discharges the document11whose content has been read onto a discharge table16B included in the document cover16.

The image forming device14forms an image represented by image data on a recording medium. Recording media are stored in storage trays19that are classified by the type or size of recording media. The image forming device14may form an image in any color on a recording medium and may form a color image or a monochrome image.

The image processing apparatus10includes, in a front portion thereof, an operation display device13that accepts an operation for executing various functions such as the copy function, the print function, and the scan function from a user.

Specifically, the operation display device13includes a reader device17that acquires information on a user who performs an operation, and an operation panel15that accepts an operation performed by the user.

For example, in response to the user bringing their employee identity card close to the reader device17, the reader device17reads identification information (referred to as a “user ID”) for uniquely identifying the user from an integrated circuit (IC) chip incorporated in the employee identity card in a contactless manner.

The operation panel15is a display having a touch panel superimposed thereon. The operation panel15displays, as an icon image, an object to be operated by the user to execute a desired function. The object may be of any type that is to be operated by the user, and includes, for example, a button, a scroll bar, a check box, and a radio button. In response to the user performing an operation on the object, the image processing apparatus10executes a process associated in advance with the content of the operation, and a response to the operation is displayed on the operation panel15.

FIGS.3A and3Billustrate an example of the operation panel15that allows detection of an operation position6of the user in a contactless manner.FIG.3Ais a sectional view of the operation panel15, andFIG.3Bis a plan view of the operation panel15when viewed in a direction facing a display surface of the operation panel15.

The operation panel15detects the position of the user's finger, that is, the operation position6, in a contactless manner. The phrase “detecting the operation position6in a contactless manner” refers to detecting the position of a user's finger in response to the user holding their finger in a position in mid-air that is above a display surface of the operation panel15and that is away from the display surface of the operation panel15in a range of the display surface of the operation panel15without pressing their finger against the display surface of the operation panel15. A mid-air space above the display surface of the operation panel15in a range of the display surface of the operation panel15is hereinafter referred to as a mid-air space “over the operation panel15” or “above the operation panel15”. The phrase “holding the user's finger over something (such as the operation panel15)” means that the user points at a position in mid-air over the operation panel15with their finger without touching the display surface of the operation panel15.

The operation panel15includes a so-called capacitive touch panel that detects the operation position6from a change in electrostatic capacitance caused by the user holding their finger over the operation panel15. In the operation panel15including such a touch panel, a change in electrostatic capacitance at a position closest to the user's finger is larger than a change in electrostatic capacitance at any other position. Accordingly, the operation panel15outputs, as the operation position6of the user, a position at which the change in electrostatic capacitance is largest within the range of the operation panel15.

To identify the operation position6of the user on the operation panel15, an operation coordinate system is defined for the operation panel15to define a detection area for contactless detection of the position of the user's finger. The operation coordinate system is represented as a three-dimensional coordinate system having any position on the operation panel15as an origin P. In the example of the operation panel15illustrated inFIGS.3A and3B, the origin Pis set at one of the vertices of the outline of the rectangular operation panel15. In the example of the operation panel15illustrated inFIGS.3A and3B, furthermore, an X axis is set along a lateral direction of the operation panel15with respect to the origin P, a Y axis is set along a longitudinal direction of the operation panel15with respect to the origin P, and a Z axis is set so as to be orthogonal to the X and Y axes. The Z-axis direction is referred to as a height direction of the operation panel15.

The operation position6of the user on the operation panel15is represented by a coordinate point (x, y), which is a combination of the coordinate value x of the X coordinate and the coordinate value y of the Y coordinate of a position at which the change in electrostatic capacitance is largest within the range of the operation panel15.

When the operation panel15displays objects, an object displayed so as to include the operation position6of the user is recognized as the object being operated by the user. In the example of the operation panel15illustrated inFIG.3B, since the operation position6of the user is included in the area of a button8in a screen30displayed on the operation panel15, the user is recognized as operating the button8. An object displayed so as to include the operation position6of the user may be hereinafter referred to as an “object corresponding to the operation position6”. The operation position6is an example of a “detected position at which an operation performed by a user has been detected” according to the present exemplary embodiment.

As illustrated inFIG.3A, the length of a perpendicular drawn from a user's finger3, which is held over the operation panel15, to the display surface of the operation panel15, that is, the distance from the user's finger3to the operation panel15in the height direction of the operation panel15, is represented by an “operation distance D”. The user's finger3is an example of a target with which the user performs operations. The target may be a body part of the user, other than the user's hand or fingers, or may be a stylus or any other tool owned by the user. As the operation distance D decreases, the change in electrostatic capacitance at the operation position6of the user increases on the operation panel15. Conversely, as the operation distance D increases, the change in electrostatic capacitance at the operation position6of the user decreases on the operation panel15. Accordingly, associating the operation distance D with the amount of change in electrostatic capacitance in advance makes it possible to obtain the operation distance D from the amount of change in electrostatic capacitance on the operation panel15.

Based on the correspondence relationship between the operation distance D and the amount of change in electrostatic capacitance, the operation panel15recognizes the operation position6of the user not only as a two-dimensional operation position6along the display surface of the operation panel15but also as a three-dimensional operation position6that takes the operation distance D into account. That is, when the operation position6of the user is represented as a three-dimensional position, the operation position6of the user is represented by a coordinate point (x, y, z) obtained by combining a coordinate value z representing the operation position6in the height direction of the operation panel15with the coordinate point (x, y). The coordinate value z is a coordinate value, on the Z axis, of a position the operation distance D away from the origin P along the Z axis.

The coordinate value z=0 means that the user is performing an operation while touching the display surface of the operation panel15with their finger. Accordingly, the image processing apparatus10also recognizes a difference in the manner of the operation of the user, such as whether the user is operating the operation panel15in a contactless manner or operating the operation panel15with their finger in contact with the operation panel15. As described above, the operation panel15supports both a contact operation in which the user performs an operation while touching the display surface of the operation panel15with their finger and a contactless operation in which the user operates the operation panel15while holding their finger over the operation panel15.

As described above, since the change in electrostatic capacitance at the operation position6of the user decreases on the operation panel15as the operation distance D increases, the operation distance D has an upper limit. If the user holds their finger over the operation panel15at a position exceeding the upper limit of the operation distance D, the electrostatic capacitance at the operation position6of the user does not change, and the operation panel15makes no response to the operation of the user.

The detection area for objects is an area in mid-air that is about 3 cm away from the operation panel15, for example. In other words, in response to the user moving the user's finger3close to a position about 3 cm from the operation panel15, the electrostatic capacitance in the corresponding object changes and a contactless input is detected. The XYZ coordinates of the position in the detection area of the user's finger3are acquired as those of the operation position6. In response to the user further moving the user's finger3to a position closer than 3 cm, the XYZ coordinates of the position are acquired.

FIG.4is a diagram illustrating an example functional configuration of the image processing apparatus10according to the present exemplary embodiment. The image processing apparatus10includes functional units, namely, a control unit20, an acceptance unit21, a display unit22, a document reading unit23, and an image forming unit24.

The acceptance unit21accepts a user ID of a user who operates the image processing apparatus10from the reader device17of the operation display device13, and also accepts the operation position6of the user on the operation panel15from the operation panel15of the operation display device13. The acceptance unit21further accepts image data from a portable storage medium connected to the terminal4of the user or the image processing apparatus10. The acceptance unit21notifies the control unit20of the user ID, the operation position6of the user, and the image data, which have been accepted.

When notified of the user ID by the acceptance unit21, the control unit20performs an authentication process to determine whether the user represented by the user ID is a user (referred to as a “registered user”) permitted to use the image processing apparatus10. When notified of the operation position6of the user on the operation panel15by the acceptance unit21, the control unit20determines whether the object displayed at the operation position6of the user is selected in the screen30displayed on the operation panel15, and executes a process associated in advance with the selected object. For example, if the object is a button8for starting the print function, the control unit20starts the print function to form an image represented by the image data accepted by the acceptance unit21on a recording medium.

Since the image processing apparatus10has the copy function, the print function, and the scan function, the control unit20includes a scan controller20A that controls the scan function, a print controller20B that controls the print function, and a copy controller20C that controls the copy function. Any one of the scan controller20A, the print controller20B, and the copy controller20C performs control in accordance with the content of the process associated with the object operated by the user. In one example, the image processing apparatus10may have a facsimile function. In this example, the control unit20includes a facsimile controller that controls the facsimile function.

When the operation performed by the user through the object is an operation related to the scan function, the scan controller20A controls the document reading unit23to implement the scan function. When the operation performed by the user through the object is an operation related to the print function, the print controller20B controls the image forming unit24to implement the print function. When the operation performed by the user through the object is an operation related to the copy function, the copy controller20C controls the document reading unit23to generate image data of the document11. Thereafter, the copy controller20C controls the image forming unit24to form an image represented by the generated image data on a recording medium.

The document reading unit23drives the document reading device12under the control of the scan controller20A and the copy controller20C to, for example, transport each of the documents11placed on the document table16A and generate image data of the transported document11.

The image forming unit24drives the image forming device14under the control of the print controller20B and the copy controller20C to, for example, transport a recording medium stored in any of the storage trays19and form an image represented by the image data on the transported recording medium.

The display unit22displays, for example, a result of the authentication process performed on the user and a result of the process executed by the control unit20in response to the operation performed by the user through the object on the operation panel15in the operation display device13in accordance with an instruction from the control unit20.

FIG.5is a diagram illustrating an example transition of the screen30displayed on the operation panel15, presenting how the screen30transitions in response to the user operating the operation panel15.

The display of the screen30on the operation panel15, which is performed by the display unit22, may also be interpreted as the display of the screen30on the operation panel15under the control unit20because the display unit22displays the screen30in accordance with an instruction from the control unit20. A mid-air space extending along the Z axis and having a bottom surface corresponding to the display range of the screen30displayed on the operation panel15is expressed as a mid-air space “over the screen30” or “above the screen30”, and a mid-air space extending along the Z axis and having a bottom surface corresponding to the display range of an object displayed in the screen30is expressed as a mid-air space “over the object” or “above the object”. Like the expression “over the operation panel15” or “above the operation panel15”, the expression “over the screen30” or “above the screen30” and the expression “over the object” or “above the object” do not mean the upper side of the screen30and the upper side of the object based on the up, down, left, and right directions in the real space, respectively, but mean a mid-air space in a direction facing the screen30and a mid-air space in a direction facing the object, respectively.

For convenience of description, screens30whose types are distinguished from each other are accompanied by different alphabet symbols associated with the types of the screens30. Screens30whose types are not distinguished from each other are collectively expressed as the “screens30” regardless of their types. Buttons8, which are an example of objects, whose types are distinguished from each other are accompanied by different alphabet symbols associated with the types of the buttons8. Buttons8whose types are not distinguished from each other are collectively expressed as the “buttons8” regardless of their types.

When it is determined that the user who performs an operation is a registered user through the authentication process, the control unit20causes a start screen30A to be displayed on the operation panel15. The start screen30A displays an instruction given to the user, such as “Please hold your hand over the screen. Let's start Touch Less!”, for example.

When the user holds their finger over the start screen30A, a cursor is displayed at the operation position6of the user on the start screen30A. In the example of the start screen30A illustrated inFIG.5, a cursor in the shape of a hand is displayed. The shape of the cursor is an example, and, for example, a circular cursor may be displayed. In response to the user holding their finger over the start screen30A, a home screen30B is displayed. The instruction given to the user in the start screen30A is also used to instruct the user how to perform an operation on the operation panel15.

The home screen30B displays, for example, buttons8for individually selecting the various functions of the image processing apparatus10, and a navigation bar9for displaying information useful for the user to perform an operation. Since the image processing apparatus10has the copy function, the print function, and the scan function, a “Copy” button8A for selecting the copy function, a “Print” button8B for selecting the print function, and a “Scan” button8C for selecting the scan function are displayed on the home screen30B. The navigation bar9displays, for example, the name of a user who has been authenticated, such as “user A”, the name of a screen being displayed on the operation panel15, such as “home”, and information for notifying the user that the operation panel15is in a contactless operation mode, such as “Touch Less”.

In response to the user holding their finger over the “Copy” button8A, the “Copy” button8A is selected. Upon selection of the “Copy” button8A, a copy screen30D is displayed on the operation panel15. The copy screen30D displays buttons8D to8G for setting copy conditions, and a copy start button8H for starting copying under the set copy conditions.

The copy screen30D illustrated inFIG.5displays, as an example of the buttons8for setting copy conditions, for example, a color mode button8D for selecting a copy color, a duplex/simplex selection button8E for selecting a double-sided (duplex) or single-sided (simplex) copy mode, an N-up button8F for selecting an image layout on a recording medium, and a number-of-copies button8G for selecting the number of copies to be made.

In response to the user holding their finger over any one of the buttons8D to8G for setting the respective copy conditions, the button8corresponding to the operation position6of the user is selected, and the screen30for setting the copy condition corresponding to the selected button8is displayed. In response to the duplex/simplex selection button8E being selected on the copy screen30D, a duplex/simplex selection screen30G for selecting a duplex or simplex copy mode is displayed on the operation panel15in such a manner as to be superimposed on the copy screen30D.

The duplex/simplex selection screen30G illustrated inFIG.5displays, for example, a duplex-to-duplex selection button8S for sequentially copying two-sided documents11on both sides of recording media, a simplex-to-duplex selection button8T for sequentially copying one-sided documents11having text and the like on either side thereof on both sides of recording media, and a simplex-to-simplex selection button8U for sequentially copying one-sided documents11having text and the like on either side thereof on either side of recording media.

In response to the user holding their finger over any one of the buttons8S to8U on the duplex/simplex selection screen30G, the button8corresponding to the operation position6of the user is selected, and a copy mode corresponding to the selected button8is set. In the example of the duplex/simplex selection screen30G illustrated inFIG.5, the duplex-to-duplex selection button8S is selected by the user.

In response to a duplex or simplex copy mode being set on the duplex/simplex selection screen30G, the copy screen30D is displayed on the operation panel15. After the setting of the copy mode, the copy mode selected on the duplex/simplex selection screen30G is displayed in the duplex/simplex selection button8E on the copy screen30D.

In the example described above, the user selects the duplex/simplex selection button8E on the copy screen30D. Also in response to the user selecting any one of the color mode button8D, the N-up button8F, and the number-of-copies button8G on the copy screen30D, a selection screen for selecting a copy condition corresponding to the selected one of the buttons8is displayed on the operation panel15in a manner similar to that for the duplex/simplex selection screen30G.

In response to the user holding their finger over the copy start button8H on the copy screen30D, the copy start button8H is selected. Upon selection of the copy start button8H, a copying process for copying the content of the documents11on recording media is executed in accordance with the set copy conditions. Before the setting of the copy conditions, the buttons8D to8G on the copy screen30D display initially set copy conditions that are set in advance.

In response to the user holding their finger over the “Print” button8B on the home screen30B, the “Print” button8B is selected. Upon selection of the “Print” button8B, a print screen30E is displayed on the operation panel15.

The print screen30E displays print information buttons8J each for displaying information on a piece of image data to be used for printing, and an all-print start button8M for starting printing of all of the pieces of image data corresponding to the respective print information buttons8J. In the example of the print screen30E illustrated inFIG.5, the print screen30E in which two pieces of image data to be used for printing are accepted is illustrated. That is, the print screen30E displays a number of print information buttons8J equal to the number of pieces of image data accepted as targets for printing from the user, each print information button8J corresponding to a corresponding one of the pieces of image data.

If the number of pieces of image data is too large to display the corresponding print information buttons8J in the print screen30E at the same time, in response to the user performing a gesture of moving their finger in an upward/downward direction of the print information buttons8J, the operation panel15detects the movement of the operation position6and scrolls the print information buttons8J. As a result, the print information buttons8J that are not displayed in the print screen30E are displayed in the print screen30E.

Each of the print information buttons8J displays a file name of image data to be used for printing and print conditions set by the user in advance for the image data. For example, when the user transmits image data from the terminal4to the image processing apparatus10, print conditions set by the user using the terminal4are displayed in the print information button8J.

In response to the user holding their finger over the all-print start button8M, the all-print start button8M is selected. Upon selection of the all-print start button8M, a printing process for printing images represented by image data on recording media is executed in accordance with the set print conditions.

In response to the user holding their finger over any one of the print information buttons8J, the print information button8J over which the finger is held is selected. Upon selection of any one of the print information buttons8J, a print edit screen30H is displayed on the operation panel15. The print edit screen30H illustrated inFIG.5is displayed, for example, in response to the user selecting the print information button8J corresponding to the image data representing “Material B.pdf”.

The print edit screen30H displays, for example, a delete button8V for deleting the image data corresponding to the selected print information button8J, a change button8W for changing a print condition of the image data corresponding to the selected print information button8J, and an individual-print start button8X for printing only the image data corresponding to the selected print information button8J. The print edit screen30H illustrated inFIG.5displays, as an example of the change button8W, a change button8W for changing the number of copies to be printed. The print edit screen30H also displays, for example, a change button8W (not illustrated) for changing any other print condition, such as the color of an image to be printed.

In response to the user holding their finger over the “Scan” button8C on the home screen30B, the “Scan” button8C is selected. Upon selection of the “Scan” button8C, a scan screen30F is displayed on the operation panel15.

The scan screen30F displays scan setting buttons8N for setting scan conditions, and a scan start button8R for starting reading of the documents11in accordance with the set scan conditions.

In response to the user holding their finger over any one of the scan setting buttons8N, the scan setting button8N corresponding to the operation position6of the user is selected, and a selection screen (not illustrated) for selecting the scan condition corresponding to the selected scan setting button8N is displayed. That is, the user sets each of the scan conditions associated with the scan setting buttons8N in the same manner as the operation of setting the copy conditions through the copy screen30D. In response to the user holding their finger over the scan start button8R, the scan start button8R is selected. Upon selection of the scan start button8R, a scanning process for converting the content of the documents11into image data is executed in accordance with the set scan conditions.

In response to the user holding their finger over the navigation bar9on the home screen30B, the navigation bar9is selected. Upon selection of the navigation bar9, a logout process of the authenticated user is performed. Then, as illustrated in a screen30C, and the navigation bar9displays an indication of completion of the logout process.

The foregoing describes an example in which any one of the buttons8is selected in response to the user holding their finger over the button8. In a contactless operation, the user's finger, which is in contact with the operation panel15, can move. If an object whose area includes the operation position6is simply set as an object selected by the user because the area includes the operation position6, another object adjacent to the object that the user is intended to operate may be incorrectly selected if the user's finger unintentionally moves. In addition, the user may pass their finger over another object not to be operated while moving the finger to above the object to be operated, and the unintended object may be incorrectly selected.

Accordingly, for example, when the user continuously holds their finger over an object for a predetermined period of time (a certain amount of time), the object over which the finger is held may be determined to be an object intentionally selected by the user. In other words, when the operation position6of the user remains located in the area of a specific object on the operation panel15for a predetermined period of time (a certain amount of time), it may be determined that the user has selected the object. The predetermined period of time may be set to 3 seconds. However, this example is not limiting. For example, the predetermined period of time may be set to a time other than 3 seconds. The method for detecting the operation position6is not limited to a detection method using the operation panel15, which is a capacitive touch panel. For example, the operation position6may be detected using a time-of-flight (ToF) camera or the like.

FIG.6is a diagram illustrating an example in which the user selects the “Copy” button8A, which is an example of an object, on the home screen30B.

In response to the user holding their finger over the “Copy” button8A, the operation position6is detected within the area of the “Copy” button8A. Such a transition from a state in which the operation position6has not been detected within the area of an object to a state in which the operation position6has been detected within the area of an object is referred to as “selection start”, “tentative selection”, or “hover”. An object has not yet been selected as long as the object is in the “selection start” state.

When the user continuously holds their finger over the “Copy” button8A and the detected operation position6remains located within the area of the “Copy” button8A for a predetermined period of time (a certain amount of time), as illustrated inFIG.6, the “Copy” button8A is selected, and the copy screen30D is displayed on the operation panel15. The confirmation of selection of an object is referred to as “selection completion” or “hold”. The completion of selection of an object is referred to as the object having been selected.

Accordingly, if the user's finger moves from over the “Copy” button8A to another location during selection start, the selection start for the “Copy” button8A is canceled. Such movement of the user's finger from over an object to another location during selection start is referred to as “deselection”. After an object is deselected, the user again continuously holds their finger over the deselected object for a predetermined period of time (a certain amount of time), thereby completing the selection of the deselected object.

Each of the objects in the screens30is associated in advance with a process to be executed in response to the selection of the object such that a copying process is executed in response to the selection of the copy start button8H. To notify the user of the processes to be executed for the respective objects, each of the objects displays, for example, information indicating the content of the process to be executed in response to the selection of the object, such as “copy” for the copy start button8H. The user understands a process associated with each of the objects by checking information indicating the content of the process to be executed in response to the selection of the object, that is, by checking an item associated with the object. As described above, the objects are displayed on the screens30in such a manner as to be associated with items each indicating the content to be processed. Accordingly, each of the objects is an example of an “item displayed on a screen” according to the present exemplary embodiment.

Next, the configuration of the substantial part of an electrical system of the image processing apparatus10will be described with reference toFIG.7. The image processing apparatus10is implemented using, for example, a computer40.

In the computer40, a central processing unit (CPU)41, a random access memory (RAM)42, a read only memory (ROM)43, a non-volatile memory44, and an input/output interface (I/O)45are connected to each other via a bus46.

The CPU41is an example of a processor configured to perform processing of the functional units of the image processing apparatus10illustrated inFIG.4. The RAM42is an example of a storage medium to be used as a temporary work area for the CPU41. The ROM43is an example of a storage medium that stores an information processing program to be executed by the CPU41. The non-volatile memory44is an example of a storage medium configured such that information stored therein is maintained even if power supply to the non-volatile memory44is shut off. Examples of the non-volatile memory44include a semiconductor memory and a hard disk. The non-volatile memory44is not necessarily incorporated in the computer40, and may be, for example, a storage medium attachable to the computer40in a removable manner, such as a memory card.

The I/O45is connected to, for example, the document reading device12, the image forming device14, an input device31, a display device32, and a communication device33.

The document reading device12and the image forming device14are devices that perform operations as described above. The input device31is a device that notifies the CPU41of an instruction from the user and a user ID of the user in response to receipt of the instruction and the user ID. Examples of the input device31include a touch panel constituting the operation panel15, and the reader device17. The display device32is a device that visually displays information processed by the CPU41. Examples of the display device32include a display constituting the operation panel15. The communication device33is connected to the communication line2and has a communication protocol for communicating with the terminals4. The devices connectable to the I/O45are not limited to the devices illustrated inFIG.7. The I/O45may be connected to a device necessary for implementing a function in accordance with the functions of the image processing apparatus10.

Since contactless operations are performed in mid-air, the user may incorrectly operate an unselected object different from an object that is in a selected state (i.e., an object for which selection is completed) if an operation for the unselected object is accepted.

To address this inconvenience, the image processing apparatus according to the present exemplary embodiment is configured to, when an object in a screen is to be operated with a user's finger in a contactless manner, accept an operation on the object that is in a selected state in response to the user's finger being located in a second region closer to the screen than a first region among a plurality of regions separated according to the distance from the screen. The first region is assigned an operation for selecting the object in the screen. The second region is assigned an operation on the object.

More specifically, as illustrated inFIGS.8and9, in the image processing apparatus according to the present exemplary embodiment, the CPU41causes one or more objects to be displayed in a print screen50on the operation panel15. In response to detecting the presence of the user's finger3in a first region R1, the CPU41accepts an operation for tentatively selecting an object in the print screen50. Then, in response to detecting the movement of the user's finger3from the first region R1to a second region R2after the object is tentatively selected, the CPU41accepts an operation for the object tentatively selected with the user's finger3in the first region R1. That is, when the CPU41detects the movement of the user's finger3from the first region R1to the second region R2after the object is tentatively selected, the object tentatively selected with the user's finger3in the first region R1is brought to selection completion (also referred to as “active state”), making a specific operation assigned to the object executable (or processable). In other words, when the user's finger3moves from the first region R1and reaches the second region R2, the tentatively selected object is brought into a selected state (selection completion).

As illustrated inFIG.9, in the present exemplary embodiment, the distance from the user's finger3to the print screen50(i.e., the operation distance D) is divided into two regions. Of the two regions, the region closer to the user's finger3is the first region R1, and the region closer to the print screen50than the first region R1is the second region R2. The first region R1and the second region R2are separated by a threshold Th. The threshold Th is a threshold for the distance from the operation panel15to the user's finger3, and is set to an appropriate value within a range not exceeding the distance at which the user's finger3is detectable. The operations to be assigned to the first region R1and the second region R2may be set as appropriate by the user.

Next, the display of the print screen50as an example of a screen on the operation panel15will be described in detail with reference toFIG.8.

FIG.8is a plan view of the print screen50according to the present exemplary embodiment.FIG.9is a diagram schematically illustrating an example of regions in mid-air above the print screen50according to the present exemplary embodiment when the print screen50is viewed from a side.

The print screen50illustrated inFIG.8displays a document selection list60, a number-of-copies setting button70, and a print start button80as an example of objects.

The document selection list60is a list of pieces of printable document data. When the document selection list60is selected, each individual piece of listed document data is selectable. In the example illustrated inFIG.8, the document selection list60has registered therein document data of documents1to5. InFIG.8, the documents1to5in the document selection list60are indicated by icon images61to65, respectively. These pieces of document data may be held in the non-volatile memory44included in the image processing apparatus10or may be held in a terminal4connected via the communication line2.

The number-of-copies setting button70is a button for inputting the number of copies to be printed. When the number-of-copies setting button70is selected, the number of copies to be printed can be set.

The print start button80is a button for executing the print function. When the print start button80is selected, a printing process is executed.

Next, an example of a contactless operation of the image processing apparatus10according to the present exemplary embodiment will be described with reference toFIGS.10to23.

In the present exemplary embodiment, the first region R1is assigned an operation for selecting an object, and the second region R2is assigned an operation on an object that is in a selected state. The operation on an object, which is assigned to the second region R2, is an operation specific to the object.

First, an operation for selecting an object will be described with reference toFIGS.10to13.FIG.10is a side view corresponding toFIG.9, illustrating a state in which the user's finger3serving as a target used for contactless operation is present in the first region R1above the print screen50.FIG.11is a front view corresponding toFIG.8, illustrating a state in which the document selection list60serving as an object in the print screen50is tentatively selected.FIG.12is a front view corresponding toFIG.8, illustrating a state in which the number-of-copies setting button70serving as another object in the print screen50is tentatively selected.FIG.13is a front view corresponding toFIG.8, illustrating a page switching operation serving as an operation for changing the print screen50to the next page.

As illustrated inFIG.10, in response to detecting the user's finger3over the print screen50, the CPU41determines the region where the user's finger3is located. In response to determining that the user's finger3is present in the first region R1, the CPU41accepts an operation for selecting an object in the print screen50. The selection operation for the first region R1is an operation of, first, moving the user's finger3to a position in the first region R1above an object to be used among a plurality of objects to tentatively select the object, and then, moving the user's finger3from the first region R1to the second region R2to change tentative selection to selection completion.

In the present exemplary embodiment, when the user's finger3is present in the first region R1, the CPU41accepts a tentative selection of any one of the document selection list60, the number-of-copies setting button70, and the print start button80displayed in the print screen50.

As illustrated inFIG.11, in response to detecting the presence of the user's finger3at a position in the first region R1above the document selection list60, the CPU41tentatively selects the document selection list60. When an object is tentatively selected, the CPU41may display a frame around the tentatively selected object. Specifically, in response to detecting the user's finger3at a position in the first region R1above an area displaying the document selection list60(i.e., the operation position6(seeFIGS.3A and3B)), the CPU41tentatively selects the document selection list60and may display a frame90around the document selection list60.

As illustrated inFIG.12, in response to detecting the movement of the user's finger3from the position in the first region R1above the document selection list60to a position in the first region R1above the number-of-copies setting button70, the CPU41tentatively selects the number-of-copies setting button70. In response to detecting the user's finger3at a position in the first region R1above an area displaying the number-of-copies setting button70, the CPU41tentatively selects the number-of-copies setting button70and may display a frame92around the number-of-copies setting button70.

In response to detecting the movement of the user's finger3from the position in the first region R1above the document selection list60to a position in the first region R1above the print start button80, the CPU41tentatively selects the print start button80. In response to detecting the user's finger3at a position in the first region R1above an area displaying the print start button80, the CPU41tentatively selects the print start button80and may display a frame (not illustrated) around the print start button80.

Further, as illustrated inFIG.13, when the print screen50continues to the next page, the CPU41may assign, to the first region R1, an operation for changing (e.g., scrolling) the print screen50to the next page. In response to detecting that the user's finger3remains located in the first region R1for a predetermined amount of time, the CPU41accepts an operation for changing the print screen50to the next page. In the example illustrated inFIG.13, the CPU41accepts a scroll operation in response to detecting the user's finger3at a position other than the positions above the objects such as the document selection list60, the number-of-copies setting button70, and the print start button80. Alternatively, the CPU41may accept an operation for changing the print screen50to the next page even in response to detecting the user's finger3at a position above any one of the objects such as the document selection list60, the number-of-copies setting button70, and the print start button80.

Then, in response to detecting the movement of the user's finger3from the first region R1to the second region R2(seeFIG.14) after the document selection list60is tentatively selected (seeFIG.11), the CPU41places the document selection list60into the active state and accepts an operation on the document selection list60. When the document selection list60is placed into the active state, each individual piece of listed document data is individually selectable. In the method for selecting a piece of listed document data, for example, when the user's finger3continuously remains located above an area of an icon image indicating any document data for a predetermined amount of time (a certain amount of time), the target document data may be selected.

In response to detecting the movement of that the user's finger3to the second region R2after the frame90is displayed around the document selection list60(i.e., tentatively selected state), as illustrated inFIG.15, the CPU41may change the color of a frame91to indicate that the selection of the document selection list60is completed. In the present exemplary embodiment, the frame90(broken line) is displayed in blue, for example, and the frame91(solid line) is displayed in red, for example. The colors of the frames are not limited to those described above. Further, the change in display is not limited to the change in color as long as the frame90is changed to the frame91, which is different in display from the frame90, when the document selection list60is brought from the tentatively selected state to the selected state. For example, the frame90and the frame91may be different in line type, line thickness, or line shape.

Further, as illustrated inFIG.16, even in response to detecting the user's finger3at a position above an unselected object, namely, the number-of-copies setting button70or the print start button80, when the user's finger3is present in the second region R2, the CPU41does not accept an operation for selecting such an unselected object.

Further, as illustrated inFIG.17, the CPU41may change the color of the unselected objects, namely, the number-of-copies setting button70and the print start button80, when the user's finger3is located in the second region R2. Changing the color of the unselected objects allows the user to recognize that the operation for selecting the objects whose color has been changed is not accepted.

Further, as illustrated inFIG.18, in response to detecting the movement of the user's finger3from the second region R2to the first region R1after the document selection list60is selected, the CPU41may again accept an operation for selecting the number-of-copies setting button70and the print start button80with the user's finger3in the first region R1. In the present exemplary embodiment, the CPU41may be configured not to cancel the selected state of the document selection list60even in response to detecting the movement of the user's finger3from the second region R2to the first region R1and configured to allow the user to continue the operation on the document selection list60in response to detecting the movement of the user's finger3from the first region R1back to the second region R2. By contrast, the CPU41may be configured to cancel the selected state of the document selection list60when the movement of the user's finger3from the second region R2to the first region R1is detected and the user's finger3is detected at a position above another object to complete the selection of the other object.FIG.19illustrates a state in which the user's finger3moves from the second region R2to the first region R1after the document selection list60is selected, and then the user's finger3moves to a position above the number-of-copies setting button70to tentatively select the number-of-copies setting button70. In response to detecting the movement of the user's finger3from the first region R1to the second region R2with the user's finger3remaining above the number-of-copies setting button70, the CPU41completes the selection of the number-of-copies setting button70to selection completion.

Further, as illustrated inFIG.20, in response to detecting the user's finger3at a position above an unselected object, namely, the number-of-copies setting button70or the print start button80, for a predetermined amount of time while the user's finger3is detected in the second region R2, the CPU41may bring the unselected object, namely, the number-of-copies setting button70or the print start button80, to the selected state and accept an operation on the number-of-copies setting button70or the print start button80, which is brought into the selected state.

As illustrated inFIG.20, furthermore, in response to detecting the movement of the user's finger3from the second region R2to the first region R1, the CPU41may accept an operation for scrolling the print screen50. When the operation of scrolling the print screen50is executed, the CPU41may cancel the selected state of the document selection list60.

Further, as illustrated inFIG.21, in response to detecting the user's finger3at a position above the print start button80, which is an unselected object, for a predetermined amount of time when the user's finger3is present in the second region R2, the CPU41may cancel the selected state of the document selection list60and brings the print start button80, which is unselected, into a selected state. Then, the CPU41may accept an operation on the print start button80brought into the selected state.

Further, as illustrated inFIG.22, in response to the user moving the user's finger3to a position above the print screen50from one side to another of the print screen50, the CPU41may divide the mid-air space from the user's finger3to the print screen50into the first region R1and the second region R2even if the distance from the user's finger3to the print screen50(i.e., the operation distance D (seeFIG.23)) is insufficient. Further, the CPU41may cause information to be displayed on the print screen50if the operation distance D is shorter than a predetermined threshold. The information to be displayed on the print screen50may be text or an image indicating that the user's finger3is too close to the print screen50.

Next, the operation of the image processing apparatus10will be described with reference toFIG.24.

FIG.24is a flowchart illustrating an example of a process based on an information processing program according to the present exemplary embodiment.

First, in response to an instruction to execute a contactless input through the operation panel15, the CPU41activates the information processing program and executes the steps described below.

Referring toFIG.24, in step S200, the CPU41determines whether the user's finger3, which is a target performing a contactless operation, has been detected over the print screen50. If the user's finger3has not been detected, the processing of step S200is repeatedly performed after a predetermined amount of time has elapsed. If the user's finger3has been detected, the process proceeds to step S202.

In step S202, the detection area of the user's finger3is checked. Specifically, the operation distance D of the user's finger3is determined.

In step S204, the CPU41determines, based on the operation distance D of the user's finger3, whether the user's finger3is present in the first region R1. If the user's finger3is present in the first region R1, the CPU41performs a control process of the first region R1, that is, a process associated with the operation assigned to the first region R1. When the control process of the first region R1is completed, the process proceeds to step S200. If the user's finger3is not present in the first region R1, the process proceeds to step S206.

In step S206, the CPU41determines whether an object in the print screen50is selected. If no object is selected, the process proceeds to step S208. On the other hand, if an object is selected, the CPU41performs a control process of the second region R2, that is, a process associated with the operation assigned to the second region R2. When the control process of the second region R2is completed, the process proceeds to step S200.

In step S208, the CPU41adjusts an area detection position of the user's finger3and then performs the control process of the first region R1. When the control process of the first region R1is completed, the process proceeds to step S200.

Next, the control process of the first region R1will be described with reference toFIG.25.

The control process of the first region R1is started during the process illustrated inFIG.24.

First, in step S300, the CPU41performs a process of detecting a contactless operation. Then, the process proceeds to step S302.

In step S302, the CPU41determines whether the user's finger3is within the first region R1. If the user's finger3is out of the first region R1, the control process of the first region R1ends, and the process returns to the process illustrated inFIG.24. On the other hand, if the user's finger3is within the first region R1, the process proceeds to step S304.

In step S304, the CPU41determines whether the operation of the user's finger3in the first region R1is a pointing movement. If the operation of the user's finger3is a pointing movement, the process proceeds to step S306. If the operation of the user's finger3is not a pointing movement, the process proceeds to step S312. The pointing movement is an operation for moving the user's finger3within the first region R1to select an object in the screen.

In step S306, the CPU41determines whether a selectable object is selected. If a selectable object is selected, the process proceeds to step S308. If a selectable object is not selected, the process proceeds to step S300.

In step S308, the CPU41determines whether a selected object is selected. If a selected object is selected, the process proceeds to step S310. If a selected object is not selected, the process proceeds to step S300.

In step S310, the CPU41performs a process of switching display of the selected object. Specifically, the CPU41displays a frame around the selected object. After that, the process proceeds to step S300.

In step S312, the CPU41determines whether the operation of the user's finger3in the first region R1is a sliding movement. If the operation of the user's finger3is a sliding movement, the process proceeds to step S314. If the operation of the user's finger3is not a sliding movement, the process proceeds to step S300.

In step S314, the CPU41determines whether a selected object is found. If no selected object is found, the process proceeds to step S318. If a selected object is found, the process proceeds to step S316.

In step S316, the CPU41deselects the selected object. Then, the process proceeds to step S318.

In step S318, the CPU41determines that the operation of the user's finger3is an operation for scrolling the print screen50, and performs a screen switching process. After that, the process proceeds to step S300.

Next, the control process of the second region R2will be described with reference toFIG.26.

The control process of the second region R2is started during the process illustrated inFIG.24.

First, in step S350, the CPU41performs a process of detecting a contactless operation. Then, the process proceeds to step S352.

In step S352, the CPU41determines whether the user's finger3is within the second region R2. If the user's finger3is out of the second region R2, the control process of the second region R2ends, and the process returns to the process illustrated inFIG.24. On the other hand, if the user's finger3is within the second region R2, the process proceeds to step S354.

In step S354, the CPU41determines whether an object other than a selectable object is selected. If an object other than a selectable object is selected, the process proceeds to step S350. If no object other than a selectable object is selected, the process proceeds to step S356.

In step S356, the CPU41determines whether any other object that is in a selected state is found. If no other object that is in a selected state is found, the process proceeds to step S358. If any other object that is in a selected state is found, the process proceeds to step S362.

In step S358, the CPU41brings an unselected object to a selected state. After that, the process proceeds to step S360.

In step S360, the CPU41performs a process specific to the object that is in a selected state. After that, the process proceeds to step S350.

In step S362, the CPU41determines whether a predetermined amount of time has elapsed since the object was selected. If the predetermined amount of time has elapsed, the process proceeds to step S364. If the predetermined amount of time has not elapsed, the process proceeds to step S350.

In step S364, the CPU41cancels the selected state of the object. After that, the process proceeds to step S358.

In the way described above, the image processing apparatus10controls the contactless operation.

In the present exemplary embodiment, as described above, when the user's finger3is present in the second region R2, the CPU41accepts an operation on an object as long as the object is selected with the user's finger3in the first region R1. Thus, the object is less likely to be operated by mistake than when, for example, an operation on an object that is not selected with the user's finger3in the first region R1is accepted while the user's finger3is located in the second region R2. The operation for completing the selection of an object is an operation for moving the user's finger3from the first region R1to the second region R2and is different from an operation for keeping the user's finger3at a certain position in the first region R1, such as a scroll operation for changing the screen to the next page. Thus, the object is less likely to be operated by mistake when the user's finger3is present in the first region R1. In addition, when an object is in a tentatively selected state, a frame is displayed around the object, which makes it easy to visually check the tentatively selected state.

In the exemplary embodiment described above, the mid-air space between the user's finger3and the print screen50is divided into two regions. However, the present disclosure is not limited to this configuration. The mid-air space between the user's finger3and the print screen50may be divided into three or more regions, for example.FIG.27illustrates three regions into which the mid-air space is divided. In a case where the space is divided into three regions including a first region R1, a second region R2, and a third region R3by a first threshold Th1, document data in the document selection list60may be selected with the user's finger3in the second region R2, and an operation on an image of the selected document data may be completed in response to the movement of the user's finger3from the second region R2to the third area R3.

An image processing apparatus has been described as an example of an information processing apparatus according to an exemplary embodiment. Exemplary embodiments may provide a program for causing a computer to execute the functions of the information processing apparatus. Exemplary embodiments may provide a non-transitory computer-readable storage medium storing such a program.

In addition, the configuration of an information processing apparatus described in the exemplary embodiment described above is an example and may be changed depending on the situation without departing from the scope of the present disclosure.

Additionally, a process flow of a program described in the exemplary embodiment described above is also an example, and any unnecessary step may be deleted, a new step may be added, or the processing order may be changed without departing from the scope of the present disclosure.

In the exemplary embodiment described above, a program is executed to implement processing according to an exemplary embodiment by a software configuration using a computer, by way of example but not limitation. The exemplary embodiment may be implemented by, for example, a hardware configuration or a combination of a hardware configuration and a software configuration.

APPENDIX

An information processing system comprising:a processor configured to:when an object in a screen is to be operated with a target in a contactless manner, accept an operation on the object that is in a selected state in response to the target being located in a second region closer to the screen than a first region among a plurality of regions separated according to a distance from the screen, the first region being assigned an operation for selecting the object in the screen, the second region being assigned an operation on the object.
(((2)))

The information processing system according to (((1))), wherein the processor is configured to display a frame around the object in response to detecting the target at a position in the first region above the object.

The information processing system according to (((2))), wherein the processor is configured to, in response to movement of the target to the second region after the frame is displayed around the object, bring the object having the frame displayed therearound into the selected state and change a color of the frame.

The information processing system according to any one of (((1))) to (((3))), whereinthe object in the screen comprises a plurality of objects, andthe processor is configured to change a color of an unselected object among the plurality of objects.
(((5)))

The information processing system according to any one of (((1))) to (((3))), whereinthe object in the screen comprises a plurality of objects, andthe processor is configured to, in response to movement of the target from the second region to the first region after an object among the plurality of objects is brought into the selected state, again accept selection of the object with the target in the first region.
(((6)))

The information processing system according to any one of (((1)) to ((3))), whereinthe object in the screen comprises a plurality of objects, andthe processor is configured to, in response to detecting the target at a position in the second region above an unselected object among the plurality of objects for a predetermined amount of time, bring the unselected object into the selected state and accept an operation on the object brought into the selected state.
(((7)))

The information processing system according to any one of (((1))) to (((6))), whereinthe processor is configured to, in response to detecting the target over the screen, divide a mid-air space from a position of the detected target to the screen into regions including the first region and the second region.
(((8)))

The information processing system according to (((7))), whereinthe processor is configured to cause information to be displayed on the screen when a distance from the position of the detected target to the screen is shorter than a predetermined threshold.
(((9)))

The information processing system according to any one of (((1))) to (((8))), whereinthe first region is assigned an operation for changing the screen, andthe processor is configured to accept the operation for changing the screen in response to detecting that the target remains located in the first region for a predetermined amount of time.
(((10)))

The information processing system according to (((9))), whereinthe processor is configured to, in response to the operation for changing the screen after the object is brought into the selected state, cancel the selected state of the object.
(((11)))

An information processing program for causing a computer to execute:when an object in a screen is to be operated with a target in a contactless manner, accepting an operation on the object that is in a selected state in response to the target being located in a second region closer to the screen than a first region among a plurality of regions separated according to a distance from the screen, the first region being assigned an operation for selecting the object in the screen, the second region being assigned an operation on the object.