Patent Description:
As a method for remotely operating an operation screen of an image processing apparatus, a method using Virtual Network Computing (VNC) with the Remote Framebuffer (RFB) protocol has been discussed. This method enables an operation screen of an image processing apparatus to be remotely operated from a personal computer (PC) or a mobile terminal. More specifically, an image processing apparatus having a VNC server function transmits screen information about an operation screen to a terminal having a VNC client function. The terminal displays the same screen as the operation screen displayed on an operation panel of the image processing apparatus, based on the received screen information. By performing operations on the screen displayed on the terminal, the user can log in to the image processing apparatus on the operation screen provided by the image processing apparatus and instruct the image processing apparatus to perform processing without operating the operation panel of the image processing apparatus.

<CIT> discusses a method for remotely operating an image processing apparatus using VNC, where the image processing apparatus enters a power saving mode if a main body operation unit of the image processing apparatus has not received an operation for a certain period of time.

An apparatus that provides a remote operation function often limits the number of apparatuses that can remotely operate the apparatus to one in order to limit the number of operators to one. In the case of remotely operating an image processing apparatus from a terminal using VNC, if a user has established a VNC connection with the image processing apparatus, the other users are unable to establish a VNC connection with the image processing apparatus. For example, suppose that a user establishes a VNC connection with the information processing apparatus on the terminal, and operates the terminal to log in to the image processing apparatus and instruct the image processing apparatus to perform print processing. In this case, if the user forgets to disconnect the VNC connection and switches an application operating as the VNC client on the terminal to a background state after the end of the processing, the other users are unable to establish a VNC connection with the image processing apparatus since the VNC connection is still established. Additionally, <CIT> discloses a method for an information processing apparatus for communicating with an image processing apparatus.

Embodiments of the present disclosure are directed to providing a method where even if a user has finished using a remote operation function of an apparatus but has not stopped the remote operation function, another user can remotely operate the apparatus.

According to a first aspect of the present disclosure, there is provided a method for an information processing apparatus as specified in claims <NUM> to <NUM>. According to a second aspect of the present disclosure, there is provided a system as specified in claims <NUM> to <NUM>. According to a third aspect of the present disclosure, there is provided an information processing apparatus as specified in claim <NUM>. According to a fourth aspect of the present disclosure, there is provided a storage medium as specified in claim <NUM>.

Further features of the present disclosure will become apparent from the following description of exemplary embodiments with reference to the attached
drawings.

Exemplary embodiments of the present disclosure will be described below with reference to the accompanying drawings. The exemplary embodiments described below are not intended to limit the disclosure, and all combinations of features described in the exemplary embodiments are not necessarily indispensable to solving means of the present disclosure.

First, a system configuration according to an exemplary embodiment of the present disclosure will be described with reference to <FIG>. In the present exemplary embodiment, a mobile terminal <NUM> and a multifunction peripheral (MFP) <NUM> will be described as examples of information processing apparatuses. However, apparatuses included in an information processing system are not limited thereto. The mobile terminal <NUM> and the MFP <NUM> are connected to a local area network (LAN) <NUM> and can exchange information with each other. The mobile terminal <NUM> is connected to the LAN <NUM> via an access point <NUM>.

The mobile terminal <NUM> has a Virtual Network Computing (VNC) client function for remotely operating the MFP <NUM>. More specifically, the mobile terminal <NUM> includes VNC client software that supports the Remote Framebuffer (RFB) protocol. The mobile terminal <NUM> can establish a VNC connection with the MFP <NUM> and display, on an operation panel <NUM> (see <FIG>), the same screen as that displayed on an operation panel (DISPLAY) <NUM> (see <FIG>) of the MFP <NUM>. The MFP <NUM> has a VNC server function. When the mobile terminal <NUM> has established a VNC connection with the MFP <NUM>, the MFP <NUM> transmits screen information to the mobile terminal <NUM> serving as the VNC client. The mobile terminal <NUM> draws a screen based on the screen information transmitted from the MFP <NUM>. The screens displayed on the operation panels (display units) <NUM> and <NUM> of the mobile terminal <NUM> and the MFP <NUM> are liked to each other so that the screens are changed based on an operation received on the drawn screen or an operation received on the MFP <NUM>.

Next, a hardware configuration of the mobile terminal <NUM> will be described with reference to <FIG>. The mobile terminal <NUM> according to the present exemplary embodiment is assumed to be an apparatus such as a smartphone or a tablet personal computer (PC). Alternatively, the mobile terminal <NUM> may be another apparatus as long as the apparatus is an information processing apparatus communicable with the MFP <NUM>.

A central processing unit (CPU) <NUM> reads control programs stored in a read-only memory (ROM) <NUM>, and performs various types of processing for controlling an operation of the mobile terminal <NUM>. The ROM <NUM> stores the control programs. A random access memory (RAM) <NUM> is used as a temporary storage area such as a main memory or a work area of the CPU <NUM>. A storage <NUM> is an embedded MultiMediaCard (eMMC), a solid-state drive (SSD), or a hard disk drive (HDD), for example. The storage <NUM> stores various types of data such as pictures and electronic documents. An operating system (OS) <NUM> (see <FIG>) and an MFP application <NUM> (see <FIG>), which will be described below, are also stored in the storage <NUM>. A real time clock (RTC) <NUM> clocks time.

While the mobile terminal <NUM> is configured so that the single CPU <NUM> performs each processing illustrated in a flowchart to be described below, any other configuration may be employed. For example, a plurality of the CPUs <NUM> can perform each processing illustrated in the flowchart in a cooperative manner.

The operation panel <NUM> includes a touch panel function capable of detecting a user's touch operation, and displays various screens provided by the OS <NUM> and the MFP application <NUM>. The user can input a desired operation instruction to the mobile terminal <NUM> by inputting a touch operation to the operation panel <NUM>.

A speaker <NUM> and a microphone <NUM> are used when the user calls another mobile terminal or a subscriber phone, for example. A camera <NUM> captures an image based on user's image capturing instructions. Pictures captured by the camera <NUM> are stored in a predetermined area of the storage <NUM>. A wireless communication unit <NUM> performs wireless communication such as wireless LAN communication.

Next, a software configuration of the mobile terminal <NUM> will be described with reference to <FIG> is a functional block diagram of software that is implemented by the CPU <NUM> reading the control programs stored in the ROM <NUM> or the storage <NUM>.

The OS <NUM> is software for controlling an entire operation of the mobile terminal <NUM>. Various applications, including the MFP application <NUM> (described below), can be installed on the mobile terminal <NUM>. The OS <NUM> exchanges information with the applications, and changes the screen displayed on the operation panel <NUM> based on instructions received from the applications. The OS <NUM> also includes a group of device drivers for controlling various pieces of hardware, and provides the applications operating on the OS <NUM> with application programming interfaces (APIs) for using the pieces of hardware. In the present exemplary embodiment, for example, a wireless LAN control unit <NUM> corresponds to the group of device drivers. The wireless LAN control unit <NUM> is a device driver for controlling the wireless communication unit <NUM>.

The MFP application <NUM> is an application installed on the mobile terminal <NUM>. The MFP application <NUM> can issue a print instruction and a scan instruction to the MFP <NUM>. As described above, the MFP application <NUM> also has the function of operating as a VNC client to display an operation screen of the MFP <NUM> and remotely operate the MFP <NUM>. While various applications other than the MFP application <NUM> can be installed on the mobile terminal <NUM>, a description thereof will be omitted here.

A configuration of the MFP application <NUM> will be described in more detail. A screen control unit <NUM> controls the screen displayed on the operation panel <NUM> via the OS <NUM>. The screen of the MFP application <NUM> is displayed on the operation panel <NUM> by the screen control unit <NUM>. The screen control unit <NUM> also receives the operation instruction input by the user via the operation panel <NUM>. A communication unit <NUM> controls wireless communication of the wireless communication unit <NUM> via the OS <NUM>, and transmits and receives information to and from an external apparatus such as the MFP <NUM>.

A device search control unit <NUM> generates search data for searching for an external apparatus to be connected to the mobile terminal <NUM>, and transmits the search data to external apparatuses on the LAN <NUM> via the wireless communication unit <NUM>. The device search control unit <NUM> receives a response from an external apparatus responding to the search. The search data may be broadcast to all the external apparatuses on the LAN <NUM> or may be transmitted to one external apparatus by specifying an address.

A print job generation unit <NUM> generates a print job. The print job generated by the print job generation unit <NUM> is transmitted to the MFP <NUM> by the wireless communication unit <NUM>. The MFP <NUM> performs printing based on the received print job.

A scan job control unit <NUM> transmits a scan instruction to the MFP <NUM> via the wireless communication unit <NUM>. The MFP <NUM> performs a scan based on the scan instruction, and transmits data (scan data) generated by the scan to the mobile terminal <NUM>. The mobile terminal <NUM> displays the received scan data. When the scan data is to be saved, the scan data is stored into an application storage unit <NUM>. The stored scan data is managed by a document management unit <NUM>.

A remote operation unit <NUM> displays a screen based on the screen information obtained from the MFP <NUM> that connects to the mobile terminal <NUM> using VNC, and receives an operation on the displayed screen. In other words, the remote operation unit <NUM> operates as the VNC client. More specifically, the remote operation unit <NUM> displays the screen based on the screen information received from the VNC server, and transmits, to the MFP <NUM>, operation information including the position and type of the operation received on the displayed screen. The MFP <NUM> changes the screen displayed on the MFP <NUM> and the screen displayed on the mobile terminal <NUM>, based on the received operation information. The application storage unit <NUM> temporarily stores various types of information generated by the MFP application <NUM>. A clocking unit <NUM> clocks time.

<FIG> illustrates an example of a hardware configuration of the MFP <NUM>. The MFP <NUM> includes a CPU <NUM> that executes programs stored in a ROM <NUM> or a mass storage device (HD) <NUM> such as a hard disk. The CPU <NUM> controls the devices connected to a system bus <NUM> in a centralized manner.

A RAM <NUM> functions as a main memory or a work area of the CPU <NUM>. An external input controller (PANELC) <NUM> controls instructions input via various hardware keys <NUM> or a touch panel (PANEL) <NUM> provided on the MFP <NUM>. A display controller (DISPC) <NUM> controls display on the display module (DISPLAY) <NUM> such as a liquid crystal display. A disk controller (DKC) <NUM> controls the mass storage device (HD) <NUM>.

An operation unit <NUM> of the MFP <NUM> will be described now with reference to <FIG>. A screen is displayed on the DISPLAY <NUM>. The hardware keys <NUM> are arranged next to the DISPLAY <NUM>. Both the touch panel <NUM> on the DISPLAY <NUM> and the hardware keys <NUM> can receive user's operations. A logout key <NUM>, one of the hardware keys <NUM>, is a button that is pressed by the user who logs in to the MFP <NUM> in order to instruct the MFP <NUM> to perform logout processing.

Returning to <FIG>, a network interface card (NIC) <NUM> bidirectionally exchanges data with an external apparatus (such as the mobile terminal <NUM>) or a file server via the LAN <NUM>. A wireless communication module (WLAN) <NUM> connects to the access point <NUM> or operates in an access point mode, thereby enabling the MFP <NUM> to operate as an access point and establish a direct wireless communication connection with the mobile terminal <NUM>. A printer <NUM> is a paper printing unit using an electrophotographic method. The printing method is not limited to the electrophotographic method. A scanner <NUM> is an image reading unit for reading an image printed on paper. The scanner <NUM> is often equipped with an automatic document feeder (ADS) (not illustrated) as an option, and can automatically read a plurality of document sheets. In some cases, the mass storage device (HD) <NUM> is used as a temporary storage location for images.

<FIG> is a block diagram illustrating a software configuration of the MFP <NUM>. The software is stored in the ROM <NUM>, for example, and executed by the CPU <NUM>.

A user interface (UI) processing unit <NUM> analyzes an input from the external input controller <NUM> or operation information received from an operation information reception unit <NUM> (described below). In a case where the screen is to be changed, the UI processing unit <NUM> generates image data and displays the image data on the DISPLAY <NUM>.

The DISPLAY <NUM> displays various screens as illustrated in <FIG>. Examples of the screens include an identification (ID) card login screen <NUM>, a keyboard login screen <NUM>, a menu screen <NUM> to be displayed during login, an error screen (not illustrated) to be displayed when an error occurs, and a counter screen (not illustrated) for displaying counter information. The DISPLAY <NUM> is hidden while the MFP <NUM> is in a sleep state.

The ID card login screen <NUM> is a screen for logging in to the MFP <NUM> using an ID card associated with user information. When the ID card is held over an ID card reader of the MFP <NUM> while the ID card login screen <NUM> is displayed, the user information is read and login processing is performed. When a change to keyboard login button <NUM> is pressed, the ID card login screen <NUM> is switched to the keyboard login screen <NUM>.

The keyboard login screen <NUM> is a screen for logging in to the MFP <NUM> by inputting the user information using a keyboard. The login processing is performed by inputting the user information into a username input field <NUM> and a password input field <NUM> using the hardware keys <NUM> or a software keyboard (not illustrated) and then pressing a login button <NUM>. When a change to ID card login button <NUM> is pressed, the keyboard login screen <NUM> is switched to the ID card login screen <NUM>.

The menu screen <NUM> is displayed after the user logs in to the MFP <NUM> via the keyboard login screen <NUM> or the ID card login screen <NUM>. The user can use various functions of the MFP <NUM> by pressing a copy button <NUM>, a scan button <NUM>, or a print button <NUM> depending on the intended use. When the user presses a logout button <NUM>, the logout processing is performed.

Returning to <FIG>, a remote operation processing unit <NUM> includes modules such as a connection processing unit <NUM>, an operation information reception unit <NUM>, and a screen transmission unit <NUM>. The remote operation processing unit <NUM> is a VNC server module that performs communication using the RFB protocol, i.e., a software module that communicates with the mobile terminal <NUM> operating as the VNC client via the LAN <NUM>. The connection processing unit <NUM> performs connection processing in a case where a connection request is received from the MFP <NUM> or the mobile terminal <NUM> serving as the VNC client. The connection processing unit <NUM> performs disconnection processing in a case where a connection disconnection request is received from the MFP <NUM> or the mobile terminal <NUM>.

The operation information reception unit <NUM> receives screen operation information from the mobile terminal <NUM> or the MFP <NUM>, and notifies the UI processing unit <NUM> of the screen operation information. The screen transmission unit <NUM> is a display control unit that performs screen display control by reading, from the RAM <NUM>, the image data generated by the UI processing unit <NUM> and transmitting the image data to the mobile terminal <NUM> serving as the VNC client.

A login processing unit <NUM> is a module for performing the login processing of the MFP <NUM>. The login processing unit <NUM> performs user authentication based on user information received from the UI processing unit <NUM>, and performs a login if the user authentication is successful. If the login processing unit <NUM> receives a logout instruction from the UI processing unit <NUM>, the login processing unit <NUM> performs the logout processing. A network control unit <NUM> is a module for controlling the NIC <NUM>. Although not illustrated in <FIG>, the MFP <NUM> also includes a printer module for controlling the printer <NUM> and a scanner module for controlling the scanner <NUM>.

<FIG> and <FIG> illustrate examples of screens displayed on the operation panel <NUM> of the mobile terminal <NUM>. When the mobile terminal <NUM> is activated, a home screen <NUM> appears. An MFP application icon <NUM> is an icon for activating the MFP application <NUM>. When the MFP application icon <NUM> is operated, the MFP application <NUM> is activated.

When the MFP application <NUM> is activated, an application home screen (with no device selected) <NUM> appears. A selected device icon <NUM> displays information about a target MFP with which the MFP application <NUM> is currently communicating and to which a print job is to be transmitted. With no target MFP selected, the selected device icon <NUM> displays "No Printer". A device search button <NUM> is used to search for the target MFP. A remote operation button <NUM> is used to establish a VNC connection with the target MFP and remotely operate the target MFP. The application home screen (with no device selected) <NUM> may also display other buttons such as a print button <NUM> for transmitting a print job to the target MFP and a button for issuing a scan instruction.

When the device search button <NUM> is pressed, a search is performed for MFPs on the LAN <NUM>, and a device search screen <NUM> appears to display results of the search. More specifically, the device search screen <NUM> displays a list of MFPs that have responded to the search. For example, selecting a search result <NUM> from the displayed list enables selection of the target MFP.

When the target MFP is selected, an application home screen (with device selected) <NUM> appears. The selected device icon <NUM> displays information about the MFP selected on the device search screen <NUM>. If the user taps on the remote operation button <NUM> with the target MFP selected, the remote operation function of the target MFP is started. The following description will be given on the assumption that the target MFP is the MFP <NUM>.

When the remote operation function is started, a device connecting screen <NUM> appears. When a hardware key display button <NUM> is pressed, a UI for operating the hardware keys <NUM> of the MFP <NUM> appears. Details thereof will be described below.

When the VNC connection is completed and the remote operation function is established, the screen displayed on the DISPLAY <NUM> of the MFP <NUM> is displayed on the operation panel <NUM>. Here, a keyboard login screen <NUM> similar to the keyboard login screen <NUM> is displayed on the operation panel <NUM>. The screen contents are the same as those of the keyboard login screen <NUM> on the MFP <NUM> except that the hardware key display button <NUM> is displayed on the operation panel <NUM>. A description thereof will thus be omitted. If the ID card login screen <NUM> is displayed on the MFP <NUM>, an ID card login screen <NUM> is displayed on the operation panel <NUM>.

When a login instruction is given on the keyboard login screen <NUM>, a menu screen (with hardware keys hidden) <NUM> similar to the menu screen <NUM> is displayed on the operation panel <NUM>. At this time, the menu screen <NUM> is also displayed on the DISPLAY <NUM>.

If the hardware key display button <NUM> is tapped, a menu screen (with hardware keys displayed) <NUM> is displayed. In other words, a software keyboard to which the functions of the respective hardware keys <NUM> of the MFP <NUM> are assigned is displayed. A hardware key UI <NUM> is a UI for operating the hardware keys <NUM> of the MFP <NUM>. If a key displayed on the hardware key UI <NUM> is tapped, information about the tapped key is transmitted to the MFP <NUM>, and the MFP <NUM> performs processing based on the tapped key. If a logout (ID) key <NUM> corresponding to the logout key <NUM> of the MFP <NUM> is tapped, the user logs out from the MFP <NUM>. If an end button <NUM> on the hardware key UI <NUM> is tapped, the remote operation function is ended, i.e., the VNC connection is ended. After the VNC connection is ended, the application home screen (with device selected) <NUM> is displayed on the operation panel <NUM>. After the VNC connection is ended, the DISPLAY <NUM> continues displaying the screen displayed before the end of the VNC connection.

If the user who has finished using the remote operation function keeps the MFP application <NUM> operating as the VNC client in a non-foreground state (for example, in a background state) without performing the operation for ending the VNC connection, the following issue arises. An apparatus having a login configuration, like the MFP <NUM>, often limits the number of external apparatuses that can establish a VNC connection with the apparatus to one in order to limit the number of operators to one. If the user forgets to perform the operation for disconnecting the VNC connection after the end of the remote operation function, the other users are unable to establish a VNC connection with the MFP <NUM>. Suppose, for example, that the user starts the remote operation function, instructs the MFP <NUM> to perform print processing or scan processing from the mobile terminal <NUM>, and then forgets to press the end button <NUM> and uses another application without pressing the end button <NUM> (which switches the MFP application <NUM> to the background state). In this case, the other users are unable to establish a VNC connection with the MFP <NUM> since the VNC connection between the mobile terminal <NUM> and the MFP <NUM> is continued. To solve this issue, the mobile terminal <NUM> performs control so that if the MFP application <NUM> serving as the VNC client is no longer in the foreground state on the mobile terminal <NUM>, another apparatus is able to establish a VNC connection with the MFP <NUM>. For example, if a predetermined time has elapsed since the transition of the MFP application <NUM> from the foreground state on the mobile terminal <NUM>, the VNC connection is to be disconnected or a "priority state" (described in details below) of the VNC connection is to be canceled. In the present exemplary embodiment, the foreground state of the MFP application <NUM> refers a state where the MFP application <NUM> is active, such as a state where the MFP application <NUM> is a user's operation target on the information processing apparatus (the mobile terminal <NUM>). In other words, the MFP application <NUM> is displayed on the information processing apparatus, and performs processing in response to receiving a user's operation, such as a tap. While the MFP application <NUM> is in the foreground state, the OS <NUM> of the information processing apparatus preferentially performs the processing of the MFP application <NUM>. The background state of the MFP application <NUM> refers to a state where the MFP application <NUM> is operating but inactive and is not a user's operation target.

<FIG> is a flowchart illustrating processing in which the mobile terminal <NUM> establishes a VNC connection with the MFP <NUM>, performs a remote operation, and then disconnects the VNC connection. The steps illustrated in the flowchart of <FIG> are processed by the CPU <NUM> loading the control programs stored in a memory such as the ROM <NUM> into the RAM <NUM> and executing the control programs.

In step S601, the MFP application <NUM> is activated when the screen control unit <NUM> detects a tap on the MFP application icon <NUM> on the home screen <NUM>. In step S602, the screen control unit <NUM> detects a tap on the device search button <NUM> on the application home screen (with no device selected) <NUM>, and performs a search for MFPs. In step S603, the screen control unit <NUM> determines whether an MFP is selected on the device search screen <NUM>. If the screen control unit <NUM> determines that an MFP is selected on the device search screen <NUM> (YES in step S603), the application home screen (with no device selected) <NUM> is switched to the application home screen (with device selected) <NUM> and the processing proceeds to step S604. If not (NO in step S603), the processing returns to step S603.

In step S604, the screen control unit <NUM> determines whether the remote operation button <NUM> is selected on the application home screen (with device selected) <NUM>. If the screen control unit <NUM> determines that the remote operation button <NUM> is selected on the application home screen (with device selected) <NUM> (YES in step S604), the processing proceeds to step S605. If not (NO in step S604), the processing returns to step S604.

In step S605, the remote operation unit <NUM> issues a connection request using the RFB protocol to the MFP selected in step S603. In step S606, the remote operation unit <NUM> performs remote operation processing.

The remote operation processing in step S606 will be described in details with reference to <FIG>. First, in step S607, the remote operation unit <NUM> determines whether the connection request using the RFB protocol is accepted. If the connection request is accepted (YES in step S607), the processing proceeds to step S609. If the connection request is not accepted (NO in step S607), the processing proceeds to step S608. Whether the connection request is accepted is determined based on response data about the connection request. In step S608, the screen control unit <NUM> displays a connection rejection error screen <NUM> (see <FIG>), and the processing ends.

In step S609, the remote operation unit <NUM> performs initialization processing based on the RFB protocol. In the initialization processing, the remote operation unit <NUM> transmits VNC client information and receives VNC server information to and from the MFP <NUM>.

In step S610, the remote operation unit <NUM> displays a screen on the operation panel <NUM> based on the screen information received from the MFP <NUM>, i.e., the VNC server. For example, if the screen displayed on the DISPLAY <NUM> is the keyboard login screen <NUM>, the keyboard login screen <NUM> is displayed on the operation panel <NUM>. The remote operation unit <NUM> displays a screen in step S610 each time screen information is transmitted from the MFP <NUM>.

In step S611, the remote operation unit <NUM> determines whether the mobile terminal <NUM> is in the priority state. If the mobile terminal <NUM> is in the priority state (YES in step S611), the processing proceeds to step S612. If not (NO in step S611), the processing proceeds to step S617. The term "priority state" refers to a state where while the mobile terminal <NUM> establishes a VNC connection with the MFP <NUM>, the other apparatuses are rejected so as not to establish a VNC connection with the MFP <NUM>. For example, the VNC connection has two modes, a first priority mode and a last priority mode. In the first priority mode, a VNC connection with the apparatus connected earlier is given priority, and any other apparatus requesting a VNC connection later is controlled not to be connected. In the last priority mode, a VNC connection with the last apparatus requesting a VNC connection is given priority. In other words, a VNC connection with the apparatus connected earlier is disconnected if another apparatus requests a VNC connection later, and a VNC connection with the other apparatus requesting a VNC connection later is established. In the case of the first priority mode, since the subsequent connection requests can be rejected upon establishment of the VNC connection, the mobile terminal <NUM> enters the "priority state" once the VNC connection has successfully been established. By contrast, in the case of the last priority mode, even when a VNC connection has been successfully established, the VNC connection is to be disconnected if a VNC connection request is subsequently received from another apparatus. Thus, in the case of the last priority mode, the following control is performed for example. The mobile terminal <NUM> enters the "priority state" after a lapse of a predetermined time from the VNC connection. While the mobile terminal <NUM> is in the "priority state", if the VNC connection is disconnected due to a subsequent VNC connection request from another apparatus, the mobile terminal <NUM> automatically establishes a VNC connection again to restore the VNC connection. This control apparently disables the apparatus attempting a VNC connection later from establishing a VNC connection. In the last priority mode, this state will be referred to as the "priority state". While the example where the mobile terminal <NUM> enters the "priority state" after a lapse of a predetermined time from the VNC connection has been described above, this is not restrictive. The mobile terminal <NUM> may enter the "priority state" if the mobile terminal <NUM> establishes a VNC connection with the MFP <NUM> and the user logs in to the MFP <NUM>.

In step S612, the remote operation unit <NUM> determines whether the MFP application <NUM> has transitioned to the background state on the mobile terminal <NUM>. If the MFP application <NUM> is in the background state (YES in step S612), the processing proceeds to step S613. If not, i.e., the MFP application <NUM> is in the foreground state (NO in step S612), the processing proceeds to step S617. If the MFP application <NUM> has transitioned to the background state, the OS <NUM> notifies the MFP application <NUM> of the transition to the background state, and the MFP application <NUM> detects the transition of the MFP application <NUM> to the background state.

In step S613, the remote operation unit <NUM> determines whether the VNC connection is in the first priority mode or the last priority mode. The remote operation unit <NUM> may inquire of the MFP <NUM> which mode the VNC connection is in. If the VNC connection is in the first priority mode (YES in step S613), the processing proceeds to step S614. If the VNC connection is in the last priority mode (NO in step S613), the processing proceeds to step S615.

In step S614, the remote operation unit <NUM> cancels the priority state. In step S620, the remote operation unit <NUM> performs processing for disconnecting the VNC connection. In the first priority mode, the mobile terminal <NUM> enters the "priority state" once a VNC connection has successfully been established. Thus, canceling the "priority state" is synonymous with disconnecting the VNC connection. In this example, if the MFP application <NUM> has transitioned to the background state, the processing immediately proceeds to steps S614 and S620. Alternatively, the processing may proceed to steps S614 and S620 after a lapse of a predetermined time. In other words, the VNC connection may be disconnected immediately after the transition of the MFP application <NUM> to the background state, or may be disconnected after a lapse of a predetermined time from the transition to the background state.

In step S615, the remote operation unit <NUM> cancels the "priority state". Canceling the "priority state" here refers to entering a state where even if another apparatus establishes a VNC connection with the MFP <NUM> later, the mobile terminal <NUM> does not perform reconnection processing for restoring the VNC connection. In other words, the mobile terminal <NUM> transitions from the "priority state" where the mobile terminal <NUM> reconnects to restore the VNC connection if another apparatus establishes a VNC connection with the MFP <NUM> to the state where the mobile terminal <NUM> does not perform reconnection processing for restoring the VNC connection even if another apparatus establishes a VNC connection with the MFP <NUM>. This enables another apparatus to establish a VNC connection with the MFP <NUM>. The processing then proceeds to step S616.

In step S616, the remote operation unit <NUM> determines whether the MFP application <NUM> has returned to the foreground state within a certain period of time. If the MFP application <NUM> has returned to the foreground state within a certain period of time (YES in step S616), the processing proceeds to step S617. If not (NO in step S616), the processing proceeds to step S620. In step S620, the remote operation unit <NUM> performs the processing for disconnecting the VNC connection. The disconnection processing refers to, for example, transmitting operation information corresponding to the end button <NUM> to the MFP <NUM> and requesting the MFP <NUM> to disconnect the VNC connection.

In step S617, the remote operation unit <NUM> determines whether an input operation is performed on the mobile terminal <NUM>. If an input operation is performed on the mobile terminal <NUM> (YES in step S617), the processing proceeds to step S618. If not (NO in step S617), the processing proceeds to step S611. In step S618, the remote operation unit <NUM> transmits operation information about the operation received on the mobile terminal <NUM> to the MFP <NUM>.

In step S619, the remote operation unit <NUM> determines whether an instruction to disconnect the VNC connection is received. If the instruction to disconnect the VNC connection is received (YES in step S619), the processing proceeds to step S620. If not (NO in step S619), the processing proceeds to step S611. The instruction to disconnect the VNC connection is given for example by pressing the end button <NUM>.

According to the above-described processing, in a case where the mobile terminal <NUM> establishes a VNC connection with the MFP <NUM>, even if the MFP application <NUM> operating as the VNC client transitions to the background state on the mobile terminal <NUM> in response to the end of processing by the user, and the user forgets to give the instruction to disconnect the VNC connection, another apparatus can establish a VNC connection with the MFP <NUM>. Furthermore, since the VNC connection between the mobile terminal <NUM> and the MFP <NUM> can be disconnected after satisfaction of a predetermined condition, i.e., a lapse of a predetermined time from the transition of the MFP application <NUM> to the background state, if another application such as a telephony application is used during the VNC connection, the mobile terminal <NUM> can maintain the VNC connection and continue operating the MFP <NUM> as long as the predetermined time has not elapsed.

The above-described processing has been described using the case where the VNC connection is disconnected if the MFP application <NUM> is switched to the background state, as an example. However, this is not restrictive, and the VNC connection may also be disconnected if the MFP application <NUM> is closed by the user. In this case, when the MFP application <NUM> is closed, the mobile terminal <NUM> transmits a VNC disconnection request to the MFP <NUM> to disconnect the VNC connection immediately, not after a lapse of a predetermined time.

Next, processing performed by the MFP <NUM> for screen linkage with the mobile terminal <NUM> using VNC will be described with reference to <FIG>. The steps illustrated in the flowchart of <FIG> are processed by the CPU <NUM> loading the control programs stored in a memory such as the ROM <NUM> into the RAM <NUM> and executing the control programs.

In step S701, the connection processing unit <NUM> determines whether a VNC connection request is received from the mobile terminal <NUM> serving as the VNC client. If a VNC connection request is received from the mobile terminal <NUM> (YES in step S701), the processing proceeds to step S702. If a VNC connection request is not received from the mobile terminal <NUM> (NO in step S701), the processing returns to step S701. In step S702, the connection processing unit <NUM> performs negotiation processing and connects to the VNC client. In step S703, the connection processing unit <NUM> performs initialization processing to receive client information and transmit server information from and to the mobile terminal <NUM>.

In step S704, the screen transmission unit <NUM> transmits screen information about the screen displayed on the DISPLAY <NUM> to the mobile terminal <NUM> serving as the VNC client.

In step S705, the operation information reception unit <NUM> determines whether an operation is received from the user. If an operation is received from the user (YES in step S705), the processing proceeds to step S706. If not (NO in step S705), the processing returns to step S705. For example, the operation information reception unit <NUM> determines that an operation is received from the user if operation information is received from the mobile terminal <NUM> or if a user's operation is detected by the PANEL <NUM> or the hardware keys <NUM> of the MFP <NUM>.

In step S706, the operation information reception unit <NUM> determines whether the operation received in step S705 is an operation to disconnect the VNC connection. If the received operation is the operation to disconnect the VNC connection (YES in step S706), the processing proceeds to step S707. In step S707, the connection processing unit <NUM> disconnects the VNC connection. If not (NO in step S706), the processing proceeds to step S708.

In step S708, processing corresponding to the operation received in step S705 is performed. Examples of the processing include processing for causing the menu screen displayed on the DISPLAY <NUM> to transition to the next page, processing for enabling the user to log in to the MFP <NUM>, and processing for enabling the user to log out from the MFP <NUM>. In step S709, the operation information reception unit <NUM> determines whether the screen displayed on the DISPLAY <NUM> is changed by the processing performed in step S708. If the screen is changed (YES in step S709), the processing proceeds to step S704. If not (NO in step S709), the processing proceeds to step S705.

By using the above-described processing, the screen displayed on the MFP <NUM> can also be displayed on the mobile terminal <NUM>. The screens displayed on the MFP <NUM> and the mobile terminal <NUM> can be changed based on an operation received by the MFP <NUM> or the mobile terminal <NUM>. The above-described processing has been described using the case where the screen displayed on the mobile terminal <NUM> is changed based on an operation received by the MFP <NUM> or the mobile terminal <NUM>. Also in a case where the screen displayed on the MFP <NUM> is changed without an operation received by the MFP <NUM> or the mobile terminal <NUM>, the MFP <NUM> similarly transmits screen information after the change to the mobile terminal <NUM> and the mobile terminal <NUM> displays the screen after the change. The processing of the above-described flowchart enables disconnection of the VNC connection in response to a VNC disconnection request from the mobile terminal <NUM>. If the user still logs in to the MFP <NUM> in disconnecting the VNC connection, the logout processing may be automatically performed. Moreover, in disconnecting the VNC connection, the MFP <NUM> may automatically perform auto-clear processing. The auto-clear processing refers to processing for restoring the displayed screen to a screen set as an initial screen.

According to the above-described exemplary embodiment, even if the user switches the MFP application <NUM> operating as the VNC client to the background state without performing the processing for disconnecting the VNC connection, another user can establish a VNC connection. In the above-described exemplary embodiment, the mobile terminal <NUM> determines whether to disconnect the VNC connection, considering the "priority state" of the mobile terminal <NUM>, and transmits the request to disconnect the VNC connection. However, this is not restrictive. The MFP <NUM> may determine whether to disconnect the VNC connection, considering the "priority state" of the mobile terminal <NUM>, and disconnect the VNC connection. In this case, the MFP <NUM> obtains, from the mobile terminal <NUM>, information indicating the transition of the MFP application <NUM> to the background state, and determines whether to disconnect the VNC connection.

In the above-described exemplary embodiment, the example where the same screen as that displayed on the MFP <NUM> is displayed on the mobile terminal <NUM> by using a VNC connection has been described. However, this is not restrictive. For example, the above-described exemplary embodiment is also applicable to a case where a screen obtained by expanding the screen on the MFP <NUM> is displayed on the mobile terminal <NUM> instead of displaying the same screen as that on the MFP <NUM>.

Claim 1:
A method for an information processing apparatus (<NUM>) for communicating with an image processing apparatus (<NUM>), the method comprising:
displaying a screen of an application, wherein the screen corresponds to a screen displayed on display means (<NUM>) of the image processing apparatus and is subjected to display control by the image processing apparatus;
transmitting an instruction to stop the display control to the image processing apparatus if a first condition that the application is no longer operating as a user's operation target on the information processing apparatus, a status of a predetermined connection between the image processing apparatus for the display control and the information processing apparatus is in a first state, and a predetermined time has elapsed since stop of operation of the application as the user's operation target on the information processing apparatus is satisfied; and
transmitting the instruction to stop the display control before elapse of the predetermined time since the stop of the operation of the application as the user's operation target on the information processing apparatus if a second condition that the application is no longer operating as the user's operation target on the information processing apparatus and the status of the predetermined connection is in a second state is satisfied.