Automatic configuration of the logical orientation of multiple monitors based on captured images

A system and method for configuring the display of a virtual workspace on multiple monitors connected to a single computing device based on images/frames captured by multiple cameras is described. A monitor orientation controller analyzes the frames to detect motion/movement within the captured scene and a corresponding centroid of the detected motion. The monitor orientation controller determines the positions of the monitors relative to each other based on the calculated centroids. Based on the relative determined positions of the monitors, the monitor orientation controller adjusts how the virtual workspace is displayed on the monitors. Other embodiments are also described.

FIELD

A system and method for configuring the display of a virtual workspace on multiple monitors connected to a single computing device based on images/frames captured by multiple cameras. Other embodiments are also described.

BACKGROUND

Computing devices, such as desktop and laptop computers, allow the simultaneous use of multiple monitors. For example, a laptop computer may utilize a first monitor integrated into the body of the laptop and a second monitor communicatively coupled to and physically positioned to the right of the laptop. The multiple monitors may be configured such that a single virtual workspace spans two or more of the multiple monitors. For instance, in the example above, a virtual workspace generated by the laptop computer may span the first and second monitors such that the first monitor displays the left portion of the workspace and the second monitor displays the right portion of the workspace. In this example configuration, as a user moves a cursor from the left portion of the workspace to the right portion of the workspace, the cursor consequently moves from the first monitor to the second monitor.

Although the above configuration may be desirable when the second monitor is on the right side of the first monitor, when first and second monitors switch positions the above configuration would be confusing to the user. Instead, when the first and second monitors switch positions the first monitor should display the right portion of the workspace and the second monitor should display the left portion of the workspace. The process for altering the display configuration of multiple monitors coupled to a single computing device to correspond to their physical orientation involves the manual adjustment of display settings.

SUMMARY

A system and method for configuring the display of a virtual workspace on multiple monitors connected to a single computing device based on images/frames captured by multiple cameras is described. In one embodiment, the cameras are integrated or otherwise collocated with each respective monitor such that captured sets of frames represent a scene in front of each respective monitor from different perspectives. A monitor orientation controller analyzes the frames to detect motion/movement within the captured scene and a corresponding centroid of the detected motion. Since the sets of frames from each camera capture the scene from different perspectives, the centroid of motion will be at a different location with each set of frames.

In one embodiment, the monitor orientation controller determines the positions of the monitors relative to each other based on the calculated centroids. For example, when a centroid in a first set of frames corresponding to a first camera and a first monitor is located to the left of a centroid in a second set of frames corresponding to a second camera and a second monitor, the monitor orientation controller will determine that the first monitor is to the right of the second monitor.

Based on the relative determined positions of the monitors, the monitor orientation controller adjusts how the virtual workspace is displayed on the monitors. In the example configuration described above in which the first monitor is positioned to the right of the second monitor, the monitor orientation controller instructs the computing device to display a right portion of the virtual workspace on the first monitor and a left portion of the virtual workspace on the second monitor. In one embodiment, the monitor orientation controller adjusts how the virtual workspace is displayed on the monitors by altering or instructing an operating system to adjust system settings on the computing device.

By automatically adjusting the display of the virtual workspace based on detected motion captured by multiple cameras attached or otherwise collocated with the monitors, the virtual workspace is properly displayed across each monitor without reliance on manual configuration by a user. Although described above in relation to two monitors, in other embodiments more than two monitors may be coupled to the computing device. In these embodiments, the system and method described herein adjusts the display of the virtual workspace on these three or more monitors in a similar fashion as described in relation to two monitors.

DETAILED DESCRIPTION

Several embodiments are described with reference to the appended drawings are now explained. While numerous details are set forth, it is understood that some embodiments of the invention may be practiced without these details. In other instances, well-known circuits, structures, and techniques have not been shown in detail so as not to obscure the understanding of this description.

FIG. 1shows a computing system1according to one embodiment of the invention. The computing system1includes a computing device2, a primary monitor3A, a secondary monitor3B, and cameras4A and4B. The primary and secondary monitors3A and3B are communicatively coupled to the computing device2such that a virtual workspace5generated by software and hardware systems of the computing device2may span across both monitors3A and3B based on images/frames captured by the cameras4A and4B.

As used herein, the virtual workspace5is a graphical user interface that displays one or more windows, icons, textual objects, and other display elements that represent applications and workflow items running on the computing device2. For example, the virtual workspace5ofFIG. 1includes a desktop5A, multiple application windows5B, and an application dock5C. As shown, the virtual workspace5is simultaneously displayed across the primary monitor3A and the secondary monitor3B such that the left portion of the virtual workspace5is shown on the primary monitor3A and the right portion of the virtual workspace5is shown on the secondary monitor3B. As a user moves a cursor across the workspace5, the cursor may move between the primary and secondary monitors3A and3B. Although shown as the left portion of the virtual workspace5being displayed on the primary monitor3A and the right portion of the virtual workspace5being displayed on the secondary monitor3B, in other embodiments the primary monitor3A may display the right portion of the workspace5and the secondary monitor3B may display the left portion of the workspace5. The virtual workspace5may change over time based on inputs from users and applications. For example, the application windows5B may be moved, closed, or new application windows5B may be opened. In one embodiment, hardware and software systems in the computing device2configure the arrangement of the virtual workspace5on the primary and secondary monitors3A and3B as will be described in greater detail below.

The computing device2may be any digital device capable of generating the virtual workspace5for display on the primary and secondary monitors3A and3B simultaneously. As shown inFIG. 1, the computing device2is a laptop computer with the primary monitor3A integrated within the casing of the computing device2. In other embodiments, the computing device2may be a desktop computer, a tablet computer, a mobile computer (e.g., a mobile telephone, a personal digital assistant, and a mobile media player), or any other similar device.

FIG. 2Ashows a component diagram of the computing device2according to one embodiment. In other embodiments, the computing device2may include additional components not shown. Each element the computing device2shown inFIG. 2Awill be described below by way of example below.

The computing device2may include one or more monitor interfaces6for communicating with the primary and secondary monitors3A and3B. In one embodiment, the monitor interfaces6transmit or facilitate the transmission of data (e.g., video and images) for updating information shown on the primary and secondary monitors3A and3B over a transmission medium. For example, a first monitor interface6A may transmit data to the primary monitor3A over a local system bus (e.g., an Accelerated Graphics Port bus, a Peripheral Component Interconnect bus, a Peripheral Component Interconnect-Express bus, and a Video Electronics Standards Association Local Bus) while a second monitor interface6B may transmit data to the secondary monitor3B over the link7. The link7may be a wired connection (e.g., High-Definition Multimedia Interface (HDMI), Digital Visual Interface (DVI), DisplayPort, Mobile High-Definition Link (MHL), and Thunderbolt link) or a wireless connection (e.g., BLUETOOTH and WiFi). In one embodiment, the data transmitted for display on the monitors3A and3B includes data representing the virtual workspace5.

In one embodiment, the monitor interfaces6may include specialized graphics processing circuitry. For example, the monitor interfaces6may include a graphics processing unit (GPU) to rapidly manipulate and alter memory to accelerate the creation of images in a frame buffer intended for output to the primary and/or secondary monitors3A and3B.

The primary and secondary monitors3A and3B are electronic visual displays for presenting one or more portions of the virtual workspace5to a user. The primary and secondary monitors3A and3B may use any display technology, including a liquid crystal display (LCD), an organic light-emitting diode (OLED) display, a cathode ray tube (CRT) display, or a plasma display panel (PDP) display. As shown inFIG. 1andFIG. 2A, the primary monitor3A may be contained within the casing of the computing device2while the secondary monitor3B is external and connected to the computing device2through the link7. In other embodiments, both the primary and second monitors3A and3B may be external and connected to the computing device2through one or more links7.

As described above, the computing device2may be a desktop computer, a tablet computer, a mobile computer (e.g., a mobile telephone, a personal digital assistant, and a mobile media player), or any other similar device. In one embodiment, the computing device2is a tablet computer such that the primary monitor3A is an integrated display within the tablet computer (i.e., the computing device2). In this embodiment, the secondary monitor3B may be a display within a separate tablet computer. For example, the primary and secondary monitors3A and3B may be displays integrated within IPAD tablet computers designed by Apple Inc. In this configuration, the primary and secondary monitors3A and3B may be placed side-by-side such that synchronized scrolling may be performed on a set of virtual windows displayed on the primary and secondary monitors3A and3B and running on separate tablet computers. In another embodiment, configuration of the primary and secondary monitors3A and3B as tablet computers in a side-by-side arrangement may allow for movement of files and data (e.g., uniform resource locators (URLs)) between applications and windows displayed on each monitor3A and3B and running on separate tablet computers. For example, files, images, and data may be dragged from a window or application displayed on primary monitor3A into a separate window or application displayed on secondary monitor3B. In another example, URLs may be copied and dragged between web browsers displayed in each of the primary and secondary monitors3A and3B and running on separate tablet computers.

Although described as including two monitors3, in other embodiments the computing system1may include more than two monitors3. For example, the computing system1may include three monitors3, where a first monitor3shows the left portion of the virtual workspace5, a second monitor3shows the center portion of the virtual workspace5, and a third monitor3shows the right portion of the virtual workspace5.

As noted above, the computing system1includes the cameras4A and4B for separately and independently capturing images/frames. The cameras4A and4B may be collocated or adjacent to the monitors3A and3B, respectively, such that captured frames represent a scene directly in front of the monitors3A and3B, respectively. For example, the cameras4A and4B may separately capture a user seated in front of the monitors3A and3B from different perspectives. In one embodiment, the computing device2includes one or more camera interfaces8for communicating with the cameras4A and4B. For example, the cameras4A and4B communicate/transmit frames of the captured scene to the camera interfaces8A and8B, respectively. The received frames may thereafter be processed to determine the physical positioning of the monitors3A and3B in relation to each other as will be described in further detail below.

The cameras4A and4B may include any type of sensor for selectively capturing two-dimensional or three-dimensional frames, including a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS). The cameras4A and4B or the camera interfaces8A and8B may include any set of digital or optical filters for de-noising, enhancing, or otherwise improving captured frames for processing. The cameras4A and4B may be connected to and/or integrated with their respective monitors3A and3B and/or the computing device2. For example, as shown inFIG. 1andFIG. 2A, the camera4A is integrated into a casing of the monitor3A and the computing device2. In this embodiment, the camera4A communicates with the camera interface8A using a local bus of the computing device2. As also shown inFIG. 1andFIG. 2A, the camera4B may be separate from the monitor3B and/or the computing device2. In this embodiment, the camera4B is connected to the computing device2using the link7. The link7may be a wired or wireless connection medium (e.g., Universal Serial Bus, FireWire, BLUETOOTH, and WiFi).

As described above, the primary monitor3A may be a display in a tablet computer while the secondary monitor3B is a display in a separate tablet computer. In this arrangement, the cameras4A and4B may be integrated on a front face of each of the tablet computers, respectively.

The computing device2may include a main system processor9and a memory unit10. In one embodiment, the user-level functions of the device2are implemented under control of the processor9that has been programmed in accordance with instructions (code and data) stored in the memory unit10. The processor9and the memory unit10are generically used here to refer to any suitable combination of programmable data processing components and data storage that conduct the operations needed to implement the various functions of the device2. The processor9may be an application processor typically found in a laptop computer, desktop computer, or a smart phone, while the memory unit10may refer to microelectronic, non-volatile random access memory. An operating system may be stored in the memory unit10, along with application programs specific to the various functions of the device2. In one embodiment, a monitor orientation controller11may be stored in the memory unit10. The monitor orientation controller11determines which portions of the virtual workspace5are displayed on each monitor3A and3B based on inputs from the cameras4A and4B.

FIG. 2Bshows a hierarchical view of the monitor orientation controller11in relation to other hardware and software layers of the computing system1. As shown, the monitor orientation controller11resides at the application layer12along with other applications operable on the computing device2. In other embodiments, the monitor orientation controller11may reside at the operating system layer13along with components that manage system configuration and resource allocation within the computing device2. The operating system layer13communicates with the firmware layer14to manage operations of integrated and peripheral hardware elements within or attached to the computing device2. For example, the operating system layer13may send commands via the firmware layer14to the cameras4A and4B located at the hardware layer15to capture frames of a scene in front of the monitors3A and3B. The firmware layer14may assist in the control of other elements at the hardware layer15, including optics and dedicated image filters.

Returning toFIG. 2A, the computing device2may also include additional input devices16for controlling operation of the computing device2by a user. For example, the input devices16may include a keyboard16A, a mouse16B, and a touch screen16C. In other embodiments, the computing device may include other input mechanisms, including a network controller (e.g., an IEEE 802.11 or 802.3 controller) or a Bluetooth controller.

FIG. 3shows a component diagram of the monitor orientation controller11according to one embodiment. Each of the elements in the monitor orientation controller11may be implemented by one or more pieces of hardware and software integrated within the computing device2and/or distributed across one or more systems and components.

In one embodiment, the monitor orientation controller11includes a motion detector17. The motion detector17receives a stream/set of frames from each of the cameras4A and4B and determines the presence of moving objects in each of the streams. For example, the camera4A may successively capture the frames shown inFIGS. 4A and 4B. In this example,FIG. 4Ashows a first frame of a user captured by the camera4A when the user's eyes are open.FIG. 4Bshows a second frame captured by the camera4A after the first frame in which the user has blinked her right eye. In this example, the motion detector17detects the movement the user's right eyelid based on the first and second frames received from the camera4A.

FIGS. 5A and 5Bshow the user as captured by the camera4B. Since the camera4B is located on or near the secondary monitor3B, the perspective of the user blinking is different. Similar to the stream of frames from the camera4A, the motion detector17detects the movement of the user's right eyelid based on the first and second frames received from the camera4B.

Motion detection may be performed using any algorithm or technique. In one embodiment, the motion detector17compares successive frames in streams from each camera4A and4B to determine the number of altered pixels between each pair of frames. Upon determining a predefined number of pixels have been altered/changed between a pair of frames from a single camera4(e.g., 100 pixels), the motion detector17signals the discovery of motion in the stream of frames. In one embodiment, pre-processing may be performed on each frame to reduce the number of false positives as a result of natural differences in frames due to varied lighting, camera flicker, and CCD dark currents.

In one embodiment, the monitor orientation controller11includes a centroid calculator18for determining a centroid for the detected motion within each set of frames from the cameras4A and4B. The centroids represent the position of motion within the sets of frames. In one embodiment, the centroid may be defined by a set of Cartesian coordinates within a frame.

FIG. 6Ashows a centroid CAcalculated based on the detected motion in the first and second frames received from the camera4A and shown inFIGS. 4A and 4B.FIG. 6Bshows a centroid CBcalculated based on the detected motion in the first and second frames received from the camera4B and shown inFIGS. 5A and 5B. The centroids CAand CBinFIGS. 6A and 6Bare located around the eyes of the user where the movement was detected (e.g., blinking of the eyes of the user). The centroids CAand CBare defined by Cartesian coordinates, which indicate the positioning of the motion relative to the borders of the captured frames.

In one embodiment, the monitor orientation controller11includes an orientation calculator19. The orientation calculator determines the positions of the primary and secondary monitors3A and3B relative to each other based on centroids corresponding to the detected motion captured by each camera4A and4B. For example, the centroid CAof the detected motion in the set of frames captured by the camera4A may be located at the coordinates (4, 5). In contrast, the centroid CBof the detected motion in the set of frames captured by the camera4B may be located at the coordinates (6, 5). As shown in this example, the centroid CAis located to the right of the centroid CB. Since the cameras4A and4B have been positioned to capture the view in front of the monitors3A and3B, respectively, the orientation calculator19can conclude that the primary monitor3A is located to the left of the secondary monitor3B.

Based on the relative determined positions of the monitors3A and3B determined by the orientation calculator19, the virtual workspace configurator20adjusts how the virtual workspace5is displayed on the primary and secondary monitors3A and3B. In the example configuration described above in which the primary monitor3A is positioned to the left of the secondary monitor3B, the virtual workspace configurator20instructs the computing device2to display a left portion of the virtual workspace5on the primary monitor3A and a right portion of the virtual workspace5on the secondary monitor3B. In one embodiment, the virtual workspace configurator20adjusts how the virtual workspace5is displayed on the primary and secondary monitors3A and3B by altering or instructing the operating system to adjust system settings on the computing device2.

By adjusting the display of the virtual workspace5based on detected motion captured by multiple cameras4A and4B attached or otherwise collocated with the monitors3A and3B, the monitor orientation controller11assists in properly displaying the virtual workspace5without reliance on manual configuration by a user. In one embodiment, the monitor orientation controller11operates continuously over time to ensure the virtual workspace5is being properly displayed on the primary and secondary monitors3A and3B based on their relative positions. Although described above in relation to two monitors3A and3B, in other embodiments more than two monitors3may be coupled to the computing device2. In these embodiments, the monitor orientation controller11adjusts the display of the virtual workspace5on these three or more monitors3in a similar fashion as described above in relation to the monitors3A and3B.

Turning now toFIG. 7, a method21for adjusting the display of the virtual workspace5on the primary and second monitors3A and3B according to one embodiment will be described. Each operation in the method21may be performed by one or more components of the computing device2.

The method21begins at operations22and23with the capturing of separate sets of frames by the cameras4A and4B. The cameras4A and4B are collocated or are adjacent to the primary and secondary monitors3A and3B, respectively. Accordingly, the sets of frames captured by each camera4A and4B, represent a scene in front of the monitors3A and3B with different perspectives. The set of frames from the camera4A are captured concurrently with the set of frames from the camera4B such that both sets of frames capture the same scene at the same point in time.

Following the capture of the two sets of frames at operations22and23, operation24attempts to detect motion in each set of frames. Motion detection may be performed using any algorithm or technique. In one embodiment, operation24compares successive frames from each camera4A and4B to determine the number of altered pixels between each pair of frames. Upon determining a predefined number of pixels have been altered/changed between a pair of frames from a single camera4(e.g., 100 pixels), operation24signals the discovery of motion in the set of frames.

Upon the detection of movement in both sets of frames from the cameras4A and4B, operation25calculates the centroid for each detected movement. The centroids represent the position of motion within the frames. In one embodiment, the centroids may be defined by a set of Cartesian coordinates within a frame as shown inFIGS. 6A and 6B. The centroids CAand CBinFIGS. 6A and 6Bare located around the eyes of the user where the motion was detected (e.g., blinking of the right eye). The centroids CAand CBin this embodiment may be defined by Cartesian coordinates, which indicate the positioning of the movement relative to the borders of the captured frames. In other embodiments, different techniques may be used to uniquely identify the location of motion captured by each camera4A and4B.

Following the calculation of centroids for each set of frames in which movement/motion was detected; operation26determines the positions of the primary and secondary monitors3A and3B relative to each other. For example, when a first centroid in a first frame corresponding to the camera4A and the monitor3A is to the right of a second centroid in a second frame corresponding to the camera4B and the monitor3B, operation26determines that the monitor3A is to the left of the monitor3B. Similarly, when a first centroid in a first frame corresponding to the camera4A and the monitor3A is to the left of a second centroid in a second frame corresponding to the camera4B and the monitor3B, operation26determines that the monitor3A is to the right of the monitor3B.

After determining the relative positions of the primary and secondary monitors3A and3B, operation27adjusts how the virtual workspace5is displayed on the primary and secondary monitors3A and3B. For example, when the primary monitor3A is determined at operation26to be to the left of the secondary monitor3B, operation27instructs the computing device2to display a left portion of the virtual workspace5on the primary monitor3A and a right portion of the virtual workspace5on the secondary monitor3B. Similarly, when the primary monitor3A is to the right of the secondary monitor3B, operation27instructs the computing device2to display a right portion of the virtual workspace5on the primary monitor3A and a left portion of the virtual workspace5on the secondary monitor3B. In one embodiment, operation27adjusts how the virtual workspace5is displayed on the primary and secondary monitors3A and3B by altering or instructing the operating system to adjust system settings on the computing device2.

By automatically adjusting the display of the virtual workspace5based on detected motion captured by multiple cameras4A and4B attached or otherwise collocated with the monitors3A and3B, the method21assists in properly displaying the virtual workspace5without reliance on manual configuration by a user. In one embodiment, the method21operates continuously over time to ensure the virtual workspace5is being properly displayed on the primary and secondary monitors3A and3B based on their relative positions. Although described above in relation to two monitors3A and3B, in other embodiments more than two monitors3may be coupled to the computing device2. In these embodiments, the method21adjusts the display of the virtual workspace5on these three or more monitors3in a similar fashion as described above in relation to the monitors3A and3B.

As explained above, an embodiment of the invention may be an article of manufacture in which a machine-readable medium (such as microelectronic memory) has stored thereon instructions which program one or more data processing components (generically referred to here as a “processor”) to perform the operations described above. In other embodiments, some of these operations might be performed by specific hardware components that contain hardwired logic (e.g., dedicated digital filter blocks and state machines). Those operations might alternatively be performed by any combination of programmed data processing components and fixed hardwired circuit components.