Controlling a multi-function device based on a user limit associated with a policy

A method is disclosed. For example, the method executed by a processor of a multi-function device (MFD) includes receiving a video image of a multi-function device (MFD), applying an object detection model to the video image to generate an annotated image with bounding boxes around persons detected in the video image, determining a number of the bounding boxes, determining that the number of bounding boxes is greater than a threshold in a policy associated with the MFD, and deactivating the MFD.

The present disclosure relates generally to multi-function devices (MFDs), and relates more particularly to policy enforcement at an MFD.

BACKGROUND

Multi-function devices (MFDs) are used to process print jobs. An MFD can perform a variety of different functions including printing, scanning, copying, faxing, and the like.

An enterprise or a company may have various policies related to use of different equipment including MFDs. The policies may be instituted for a variety of different reasons. For example, the policies may be instituted for health reasons, for cost savings, for production efficiency, and the like.

SUMMARY

According to aspects illustrated herein, there are provided a method and a non-transitory computer readable medium for controlling an MFD based on user limit associated with a policy. One disclosed feature of the embodiments is a method, executed by a processor of the MFD, that comprises receiving a video image of a multi-function device (MFD), applying an object detection model to the video image to generate an annotated image with bounding boxes around persons detected in the video image, determining a number of the bounding boxes, determining that the number of bounding boxes is greater than a threshold in a policy associated with the MFD, and deactivating the MFD.

Another disclosed feature of the embodiments is a non-transitory computer-readable medium having stored thereon a plurality of instructions, the plurality of instructions including instructions which, when executed by a processor, cause the processor to perform operations to receive a video image of a multi-function device (MFD), apply an object detection model to the video image to generate an annotated image with bounding boxes around persons detected in the video image, determine a number of the bounding boxes, determine that the number of bounding boxes is greater than a threshold in a policy associated with the MFD, and deactivate the MFD.

DETAILED DESCRIPTION

The present disclosure broadly discloses a method and apparatus to control an MFD based on a user limit associated with a policy. As discussed above, an enterprise or a company may have various policies related to use of different equipment including MFDs. The policies may be instituted for a variety of different reasons. For example, the policies may be instituted for health reasons, for cost savings, for production efficiency, and the like.

An example of a health policy may be to limit a number of people at the MFD. For example, for health reasons, some policies may want users to keep away from one another to prevent spread of a viral disease. Thus, a policy may be implemented at the MFD to control operation of the MFD based on a number of people at the MFD. For example, the MFD may be deactivated when a number of people greater than a threshold is detected at the MFD. The policy may also generate alarms and notifications to administrators to clean the MFD in between uses and reactivate the MFD.

One embodiment of the present disclosure uses continuous video monitoring and analysis to detect a number of people at the MFD. An application layer of the MFD may be modified to include instructions that deactivate the MFD when the video image analysis detects more than one person within a field of view of the video camera. Once the number of people are within the threshold associated with the policy, the MFD may be reactivated for use.

In one embodiment, artificial intelligence loaded onto an edge device may be used to perform the video analysis. A counting server may also be deployed to count the number of bounding boxes identified by the edge device. As a result, the computationally intensive video analysis may be offloaded from the processor of the MFD and prevent interference with processing of print jobs, scan jobs, copy jobs, and the like.

FIG. 1illustrates an example network100of the present disclosure. In one embodiment, the network100may include an internet protocol (IP) network102in communication with an MFD108. Although a single MFD108is illustrated inFIG. 1, it should be noted that any number of MFDs108may be deployed. In addition, when multiple MFDs108are deployed, the MFDs108may be located within a single location or building or distributed across different geographic locations and/or office buildings. The MFD108may be any type of device that can perform print functions, copying functions, scanning functions, faxing functions, scan and email functions, and the like.

In one embodiment, the IP network102may include an application server (AS)104and a database (DB)106. Although a single AS104and single DB106are illustrated inFIG. 1, it should be noted that any number of application servers and databases may be deployed in the IP network102. The AS104and the DB106may be operated by a service provider that manages the operation and maintenance of the MFDs108deployed at different locations and/or different customer sites.

It should be noted that the IP network102has been simplified for ease of explanation. For example, the IP network102may include additional network components that are not shown. For example, the IP network102may include additional access networks, a gateway, a firewall, various network elements, and the like.

In one embodiment, the AS104may be a counting server that may be used to count a number of persons based on analysis of a video image with bounding boxes116that is received from the MFD108. The AS104may also be a web server that may host a website. The website may allow a user to view the video image with bounding boxes116.

In one embodiment, the DB106may be a storage server that stores various information associated with the MFD108, various user profiles, instructions associated with hosting a web page, and the like. The AS104may be communicatively coupled to the DB106to retrieve information from the DB106and/or to execute instructions stored in the DB106. Although the AS104and the DB106are shown as separate devices, the AS104and the DB106may be deployed as a single device.

In one embodiment, the network100may also include an edge device110and a video camera112. The edge device110and the video camera112may be communicatively coupled to the MFD108. The edge device110may apply an object detection model that is trained to identify portions of frames of a video image that may look like a person. The object detection model may then draw bounding boxes around the objects that look like a person to generate an annotated image with bounding boxes116. The annotated image with bounding boxes116may then be transmitted to the AS104to count the number of bounding boxes116.

In one embodiment, the video camera112may be any type of device that can continuously capture video images. The video camera112may be a red, green, blue (RGB) video camera, a black and white camera, an infrared camera, and the like.

In one embodiment, the video camera112may have a motion detector to activate when motion is detected to save power and processing resources. For example, when a person or persons move in front of the video camera112, the video camera112may begin recording to capture video images. When the person leaves and no motion is detected for a predetermined amount of time, the video camera112may be deactivated to stop recording video images.

The video camera112may be located by the MFD108to continuously capture video images of the MFD108. In one embodiment, the video camera112may be set at a distance such that a field of view of the video camera112is approximately equal to a desired distance.

For example, certain health policies may suggest that people be distanced six feet apart. Thus, the video camera112may be set at a distance that captures a field of view that is approximately equal to six feet of width and includes the MFD108.

In one embodiment, the MFD108may be associated with a policy that indicates that the MFD108should be deactivated if more than one person is detected around the MFD108. Thus, the video images captured by the video camera112may be analyzed by the edge device110to draw bounding boxes. The annotated image with bounding boxes116may be transmitted to the AS104to have the number of bounding boxes counted. The number of bounding boxes may be equivalent to a number of people detected within the field of view of the video camera112. The number of bounding boxes may be transmitted back to the MFD108. If the number of bounding boxes is greater than one, then the MFD108may determine that there are two or more people that are closer than six feet (e.g., since the video camera112is set to a distance that captures a field of view of approximately six feet wide). In response, the MFD108may execute an action, as discussed in further details below (e.g., deactivate, generate a warning, generate a notification that is transmitted, and the like). Thus,FIG. 1illustrates an example of how the MFD108may be controlled based on a user limit associated with a policy of the MFD108.

FIG. 1illustrates an example of video images being analyzed by the edge device110and the AS104. The use of the edge device110and the AS104may prevent processing capabilities of the MFD108from being overwhelmed from continuous video analysis. Video analysis can be computationally intensive. Thus, the edge device110and the AS104may allow the processing resources of the MFD108to be dedicated to executing various job functions (e.g., print jobs, scan jobs, copy jobs, and the like). However, it should be noted that as processing capabilities of the MFD108are improved, the video analysis may be performed locally by the MFD108without using the edge device110and the AS104.

FIG. 2illustrates a block diagram of an example MFD108of the present disclosure. In one embodiment, the MFD108may include a processor202, a memory204, a communication interface212, a universal serial bus (USB) interface210, and a display218. It should be noted that the MFD108has been simplified for ease of explanation and may include additional components that are not shown. For example, the MFD108may include an optical scanner, a print path, a printhead, a paper tray, various finishing modules, an output tray, and the like.

In one embodiment, the processor202may be communicatively coupled to the memory204, the communication interface212, the USB interface210, and the display218. The processor202may execute instructions stored in the memory204to perform the functions described herein.

In one embodiment, the memory204may be a non-transitory computer readable medium, such as a random access memory (RAM), a read only memory (ROM), a solid state drive, a hard disk drive, and the like. In one embodiment, the memory204may store a user limit policy206and video images208. In one embodiment, the user limit policy206may include a threshold for a number of persons or users that may be allowed at the MFD108. For example, the threshold may be one person, three people, and the like.

In one embodiment, the video images208may be temporarily stored to be analyzed by the edge device110. The video images208may be deleted after being analyzed to preserve the privacy of people who may be using the MFD108. For example, in some instances the users may be recorded at the MFD with or without their consent. As a result, the images may be used to analyze a number of people without identifying who the users are in the video image to prevent violation of any privacy rules and/or laws. In one embodiment, the portion of the memory204that temporarily stores the video images208may be a circular buffer that does not permanently store the video images208. For example, the video images208are not stored on a hard disk drive or in the cloud.

In one embodiment the memory204may also store an application or instructions that are executed by the processor202to perform the functions that are described herein. For example, the MFD108may be modified at the application layer to include an application that controls operation of the MFD108based on the number of users that are detected by the MFD108in the video images that are captured by the video camera112. An example application layer protocol may include an extensible interface protocol (EIP) or other proprietary application layer protocols.

In one embodiment, the communication interface212may be a wired or wireless communication interface. The communication interface214may be a WiFi radio, an Ethernet connection interface, and the like. The communication interface214may establish a communication path with the AS104to exchange the video image or annotated image with bounding boxes116, other types of data, notifications, and the like. As noted above, in one embodiment, the AS104may also transmit back to the MFD108the number of bounding boxes that are counted via the communication interface212.

In one embodiment, the display218may be a graphical user interface (GUI). In one embodiment, the display218may be a touch screen interface that allows users to make selections by touching the display218. In one embodiment, the display218may have a keyboard or physical buttons that allow a user to make selections shown on the display218.

In one embodiment, the USB interface210may provide interfaces to electrically connect various devices such as the edge device110and the video camera112. Although a single edge device110is illustrated inFIGS. 1 and 2, it should be noted that any number of edge devices110may be connected to the USB interface210of the MFD108.

In one embodiment, the edge device110may include a central processing unit (CPU)214and an object detection model216. The object detection model216may be applied to the video images208that are temporarily stored in the memory204and captured by the video camera112. The object detection model216may be trained to analyze frames of the video images, detect a desired object (e.g., a person in the present disclosure) with the frames of video, and draw bounding boxes around the detected objects within a frame of video. An example of the object detection model216may be the OpenVINO Toolkit for person detection.

In one embodiment, the edge device110may be a neural network stick. The neural network stick may be a USB stick that has artificial intelligence that can include neural networks to perform deep learning and/or be trained to perform various functions (e.g., apply the pre-trained object detection model216). For example, the CPU214in the neural network stick may be a vision processing unit (VPU) that can offload much of the video analysis processing from the MFD108.

As noted above, the edge device110may analyze video images captured by the video camera112.FIG. 3illustrates an example image300that is analyzed to detect a number of people at the MFD108. In one embodiment, the image300may be a single frame of video within a sequence of frames of the video image captured by the video camera112.

As noted above, the field of view of the image300may be approximately equivalent to a desired distance associated with the user limit policy206. For example, the user limit policy206may set a threshold of one person within a 6 foot distance around the MFD108.

In one embodiment, the object detection model216may be applied to the image300to detect people. The object detect model216may be executed by the edge device110to detect people1141and1142. Respective bounding boxes3021and3022may be drawn around the detected people1141and1142. After the bounding boxes3021and3022are drawn, the image may be saved as an annotated image. The annotated image may be the video image with bounding boxes116that is transmitted to the AS104.

In one embodiment, the image300with the bounding boxes3021and3022may also be shown in a webpage hosted by the AS104. The AS104may count the number of bounding boxes3021and3022. In one embodiment, an analyst viewing the web page hosted by the AS104and viewing the image with bounding boxes116on the web page may count the number of bounding boxes3021and3022. In the present example, the number of bounding boxes3021and3022is two.

The AS104may transmit the number of bounding boxes that are detected back to the MFD108. The MFD108may then compare the number to the threshold associated with the user limit policy206. Using the example above, the threshold may be set to one person in the user limit policy206. The number of bounding boxes that is detected may be compared to the threshold to determine that the number of bounding boxes is greater than the threshold. As a result, the MFD108may execute an action to enforce the user limit policy206.

In one embodiment, the processor202of the MFD108may deactivate the MFD108. In addition, the processor202may generate a notification that may be shown on the display218. The notification may indicate that more than one person was detected at the MFD108and notify the users that the MFD108cannot be reactivated until only a single person is detected at the MFD108.

In one embodiment, the MFD108may generate a message (e.g., a text message, an email, and the like) that may be transmitted to an endpoint device of an administrator (e.g., a smart phone, a laptop computer, a tablet computer, a desktop computer, and the like). The message may include a date and time of when the user limit policy206is violated, an identification of the MFD108, a location of the MFD108, and the like. Thus, the administrator may re-educate employees on the user limit policy206, determine if additional MFDs108should be deployed to help maintain the user limit policy206, and the like.

In one embodiment, the administrator may confirm that people leave the MFD108until a single user is at the MFD108and then provide an authorization signal back to the MFD108. The administrator may also confirm that the MFD has been cleaned. The MFD108may then reactivate when a single person is detected based on continuous analysis of the video images captured by the video camera112or when the authorization signal is received from the administrator.

FIG. 4illustrates a flow chart of an example method400for controlling an MFD based on user limit associated with a policy of the present disclosure. In one embodiment, the method400may be performed by the MFD102or by an apparatus, such as the apparatus500illustrated inFIG. 5and discussed below.

In one embodiment, the method400begins at block402. At block404, the method400receives a video image of an MFD. For example, a video camera may be located near the MFD to capture video images of the MFD. In one embodiment, the video camera may be placed a distance that is a function of a desired distance to be captured by the field of view of the camera. The desired distance may be a function of a distance of a health policy. For example, some health policies may indicate that people should stay six feet apart. Thus, the video camera may be placed at a distance that allows a six foot width to be captured within the field of view of the camera.

At block406, the method400applies an object detection model to the video image to generate an annotated image with bounding boxes around persons detected in the video image. For example, each frame of the video image may be analyzed. A frame of the video image may be used to generate the annotated image with bounding boxes. In one embodiment, the object detection model may be applied by the MFD.

In one embodiment, to offload the processing from the MFD, an edge device may be used to apply the object detection model. In one embodiment, the edge device may be a neural network stick that is connected to the MFD via a USB connection.

At block408, the method400determines a number of the bounding boxes. In one embodiment, the MFD may count the number of bounding boxes locally.

In one embodiment, an annotated image that includes the drawn bounding boxes may be transmitted to a counting server to offload the processing from the MFD. The counting server may count the number of bounding boxes in the annotated image. In one embodiment, the counting server may also be a server that hosts a webpage that displays the annotated image to an analyst who may be monitoring the MFD. The number of bounding boxes that is counted may be transmitted back to the MFD and received by the MFD.

At block410, the method400determines that the number of bounding boxes is greater than a threshold in a policy associated with the MFD. In one embodiment, the number of bounding boxes may be equal to a number of people that is detected in the video image. The number of bounding boxes may be compared to the threshold to determine if the policy associated with the MFD is violated. When the policy is violated the MFD may execute one or more actions in response to the violation of the policy.

At block412, the method400deactivates the MFD. In one embodiment, one of the actions may be to deactivate the MFD. The MFD may be deactivated until a single person is detected at the MFD. For example, the blocks404to410may be repeated continuously until the number of bounding boxes is less than or equal to the threshold to reactivate the MFD. To illustrate, additional video images may be captured and analyzed. The object detection model may be applied to the subsequently received video images to generate a second annotated image with bounding boxes around persons that are detected in the video image. A number of the bounding boxes in the second annotated image may be determined and compared to the threshold. If the number of bounding boxes is less than the threshold, the MFD may be activated.

In one embodiment, a notification may be shown on a display of the MFD explaining why the MFD is deactivated or refusing to execute a job request. For example, the notification may explain that the number of users at the MFD violates the policy associated with the MFD.

In one embodiment, the action may include generating and transmitting a message or notification to an endpoint device of an administrator. The administrator may confirm that the additional people near the MFD have left and a single user remains at the MFD. In one embodiment, the administrator may confirm that the MFD has been cleaned before the MFD is used again. The administrator may then transmit an authorization signal when a single person remains at the MFD and/or the MFD was cleaned. The MFD may receive the authorization signal and reactivate in response to the authorization signal.

In one embodiment, the video images may be deleted after each cycle of the method400is performed. In other words, the video images may be deleted after being analyzed. The video images may be deleted to preserve privacy of users at the MFD. In one embodiment, each frame of the video image may be deleted after being analyzed. For example, the frame captured by the video camera and the annotated images may be deleted frame after frame during the analysis process. At block414, the method400ends.

FIG. 5depicts a high-level block diagram of a computer that is dedicated to perform the functions described herein. As depicted inFIG. 5, the computer500comprises one or more hardware processor elements502(e.g., a central processing unit (CPU), a microprocessor, or a multi-core processor), a memory504, e.g., random access memory (RAM) and/or read only memory (ROM), a module505for a method for controlling an MFD based on user limit associated with a policy, and various input/output devices506(e.g., storage devices, including but not limited to, a tape drive, a floppy drive, a hard disk drive or a compact disk drive, a receiver, a transmitter, a speaker, a display, a speech synthesizer, an output port, an input port and a user input device (such as a keyboard, a keypad, a mouse, a microphone and the like)). Although only one processor element is shown, it should be noted that the computer may employ a plurality of processor elements.