Configuration and operation of display devices including content curation

Content management includes receiving an image of one or more display devices with respective first content displayed. For at least one display device, a display device identification (ID) is retrieved. A change to the first content displayed at the at least one display device is caused, based on the display device ID of the at least one display device, upon interaction with the image.

TECHNICAL FIELD

One or more embodiments generally relate to display devices and, in particular, to configuring and controlling display devices including content curation.

BACKGROUND

As the ecosystem of devices and displays grows, there is often a need for users to transfer content from one device to the other. In spite of the availability of communication systems such as BLUETOOTH® and Wi-Fi, the interaction interfaces used to achieve these operations continue to be very unintuitive. Typically, a user would need to find and remember the name of the display, its representation in a network/network identification (ID), look it up from a list of names and select that name to connect to the display. This issue is compounded when the user has to curate a cluster of displays, making it very hard to know which display they are currently working with.

SUMMARY

One or more embodiments generally relate to configuring and controlling display devices including content curation. In one embodiment, a method provides for content management that includes receiving an image of one or more display devices with respective first content displayed. For at least one display device, a display device identification (ID) is retrieved. A change to the first content displayed at the at least one display is caused, based on the display device ID of the at least one display device, upon interaction with the image.

In another embodiment, a device for content management includes a memory storing instructions and at least one processor executing the instructions including a process configured to: receive an image of one or more display devices with respective first content displayed; for at least one display device, retrieve a display device ID; and cause a change to the first content displayed at the at least one display device, based on the display device ID of the at least one display device, upon interaction with the image.

In one embodiment non-transitory processor-readable medium that includes a program that when executed by a processor perform a method comprising: receiving an image of one or more display devices with respective first content displayed. For at least one display device, a display device ID is retrieved. A change to the first content displayed at the at least one display is caused, based on the display device ID of the at least one display device, upon interaction with the image.

These and other aspects and advantages of one or more embodiments will become apparent from the following detailed description, which, when taken in conjunction with the drawings, illustrate by way of example the principles of the one or more embodiments.

DETAILED DESCRIPTION

One or more embodiments provide for easy content curation for a cluster of display devices (e.g., tablets, monitors, digital photo display frames, computing displays, television displays, projected displays, etc.) using computer vision and augmented reality (AR) based technologies in a visual and spatially intuitive manner. The embodiments provide processing for saving and retrieving virtual representations of a cluster of displays and curate them locally or remotely using a mobile device (e.g., smart phone, tablet, wearable device, portable computing device, etc.).

One or more embodiments provide a unique way of visually identifying and directly interacting with a display or a cluster of displays, that results in intuitive interactions. Additionally, advanced user interactions may be employed, such as drag and drop of selected content directly on to visual representations of displays instead of textual representations, such as network ID, BLUETOOTH® ID, etc. or of symbolic representations such as folders or icons.

One or more embodiments provide spatially intelligent image editing and curation of displays (where image means both still images—photo, and moving images—video). Images may be cropped to aspect ratios, that better match that of the displays that showcase them. Single images may be processed to be spread across multiple displays in a spatially coherent manner. Selective display of certain important regions/features of an image may be displayed on selected displays. Special effects, such as filters, may be assigned to specific displays through spatial interactions.

One or more embodiments provide for saving and retrieving virtual representations of a cluster of displays as virtual walls, which may be used to curate a cluster of displays even when geographically removed from the displays, while retaining the familiar user experience. Additionally, processing is provided to group clusters of displays into various groups and specify each group's properties such as collective behavior, content to be displayed, group special effects, etc. Dynamic reassignment of a particular display from one group to another is provided and exceptions are available to group behavior.

One or more embodiments provide for easy planning, arrangement, alignment and mounting of a cluster of displays using computer vision and AR based technologies in a visual and spatially intuitive manner. Partial or full blending of displays with the environment is provided by mimicking their features such as color, texture, material and patterns. Processing provides for measurements, calculations and for suggestions for possible arrangements of displays along with specific measurements for placement of the displays. Immediate feedback about positioning and alignment of the displays is provided, and displays may be visually pointed to for corrections, which guides users to get closer to the desired configuration desired.

Based on the ambience of the display cluster, such as color of the physical wall, available illumination, difference in color palette between the images in the display cluster, one or more embodiments provide for suggestions of suitable image filters that can enhance present images in the display in a visually pleasant manner. This ambient information may be used to suggest photos and artwork, either from the local collection or from curated services that best match the ambience of the room in which the display is placed.

One or more embodiments provide information for a far away or more ideal viewpoint of the cluster of displays, even when standing near or to their side by using perspective correction. The ambient information (such as color, texture and pattern of physical wall) is used to blend in with the environment, partially or fully by mimicking the details of a wall in certain areas of the display. Emulation of various physical mats of analog picture frames on digital picture frames is provided.

FIG.1is a schematic view of a communications system10, in accordance with one embodiment. Communications system10may include a communications device that initiates an outgoing communications operation (transmitting device12) and a communications network110, which transmitting device12may use to initiate and conduct communications operations with other communications devices within communications network110. For example, communications system10may include a communication device that receives the communications operation from the transmitting device12(receiving device11). Although communications system10may include multiple transmitting devices12and receiving devices11, only one of each is shown inFIG.1to simplify the drawing.

Any suitable circuitry, device, system or combination of these (e.g., a wireless communications infrastructure including communications towers and telecommunications servers) operative to create a communications network may be used to create communications network110. Communications network110may be capable of providing communications using any suitable communications protocol. In some embodiments, communications network110may support, for example, traditional telephone lines, cable television, Wi-Fi (e.g., an IEEE 802.11 protocol), Bluetooth®, high frequency systems (e.g., 900 MHz, 2.4 GHz, and 5.6 GHz communication systems), infrared, other relatively localized wireless communication protocol, or any combination thereof. In some embodiments, the communications network110may support protocols used by wireless and cellular phones and personal email devices (e.g., a Blackberry®). Such protocols may include, for example, GSM, GSM plus EDGE, CDMA, quadband, and other cellular protocols. In another example, a long range communications protocol can include Wi-Fi and protocols for placing or receiving calls using VOIP, LAN, WAN, or other TCP-IP based communication protocols. The transmitting device12and receiving device11, when located within communications network110, may communicate over a bidirectional communication path such as path13, or over two unidirectional communication paths. Both the transmitting device12and receiving device11may be capable of initiating a communications operation and receiving an initiated communications operation.

The transmitting device12and receiving device11may include any suitable device for sending and receiving communications operations. For example, the transmitting device12and receiving device11may include mobile telephone devices, television systems, cameras, camcorders, a device with audio video capabilities, tablets, wearable devices, and any other device capable of communicating wirelessly (with or without the aid of a wireless-enabling accessory system) or via wired pathways (e.g., using traditional telephone wires). The communications operations may include any suitable form of communications, including for example, voice communications (e.g., telephone calls), data communications (e.g., e-mails, text messages, media messages), video communication, or combinations of these (e.g., video conferences).

FIG.2shows a functional block diagram of an architecture system100that may be used for configuring and controlling display devices 1-N140(e.g., tablets, monitors, digital photo display frames, computing displays, television displays, projected displays, etc.; where N is ≥0) including content curation using an electronic device120. Both the transmitting device12and receiving device11may include some or all of the features of the electronics device120. In one embodiment, the electronic device120may comprise a display121, a microphone122, an audio output123, an input mechanism124, communications circuitry125, control circuitry126, a camera128, a visual app129(for controlling display device content curation, management, arrangement, etc.), and any other suitable components. In one embodiment, applications 1-N127are provided and may be obtained from a cloud or server130, a communications network110, etc., where N is a positive integer equal to or greater than 1.

In one embodiment, all of the applications employed by the audio output123, the display121, input mechanism124, communications circuitry125, and the microphone122may be interconnected and managed by control circuitry126. In one example, a handheld music player capable of transmitting music to other tuning devices may be incorporated into the electronics device120.

In one embodiment, the audio output123may include any suitable audio component for providing audio to the user of electronics device120. For example, audio output123may include one or more speakers (e.g., mono or stereo speakers) built into the electronics device120. In some embodiments, the audio output123may include an audio component that is remotely coupled to the electronics device120. For example, the audio output123may include a headset, headphones, or earbuds that may be coupled to communications device with a wire (e.g., coupled to electronics device120with a jack) or wirelessly (e.g., Bluetooth® headphones or a Bluetooth® headset).

In one embodiment, the display121may include any suitable screen or projection system for providing a display visible to the user. For example, display121may include a screen (e.g., an LCD screen) that is incorporated in the electronics device120. As another example, display121may include a movable display or a projecting system for providing a display of content on a surface remote from electronics device120(e.g., a video projector). Display121may be operative to display content (e.g., information regarding communications operations or information regarding available media selections) under the direction of control circuitry126.

In one embodiment, input mechanism124may be any suitable mechanism or user interface for providing user inputs or instructions to electronics device120. Input mechanism124may take a variety of forms, such as a button, keypad, dial, a click wheel, or a touch screen. The input mechanism124may include a multi-touch screen.

In one embodiment, communications circuitry125may be any suitable communications circuitry operative to connect to a communications network (e.g., communications network110,FIG.1) and to transmit communications operations and media from the electronics device120to other devices within the communications network. Communications circuitry125may be operative to interface with the communications network using any suitable communications protocol such as, for example, Wi-Fi (e.g., an IEEE 802.11 protocol), Bluetooth®, high frequency systems (e.g., 900 MHz, 2.4 GHz, and 5.6 GHz communication systems), infrared, GSM, GSM plus EDGE, CDMA, quadband, and other cellular protocols, VOIP, TCP-IP, or any other suitable protocol.

In some embodiments, communications circuitry125may be operative to create a communications network using any suitable communications protocol. For example, communications circuitry125may create a short-range communications network using a short-range communications protocol to connect to other communications devices. For example, communications circuitry125may be operative to create a local communications network using the Bluetooth® protocol to couple the electronics device120with a Bluetooth® headset.

In one embodiment, control circuitry126may be operative to control the operations and performance of the electronics device120. Control circuitry126may include, for example, a processor, a bus (e.g., for sending instructions to the other components of the electronics device120), memory, storage, or any other suitable component for controlling the operations of the electronics device120. In some embodiments, a processor may drive the display and process inputs received from the user interface. The memory and storage may include, for example, cache, Flash memory, ROM, and/or RAM/DRAM. In some embodiments, memory may be specifically dedicated to storing firmware (e.g., for device applications such as an operating system, user interface functions, and processor functions). In some embodiments, memory may be operative to store information related to other devices with which the electronics device120performs communications operations (e.g., saving contact information related to communications operations or storing information related to different media types and media items selected by the user).

In one embodiment, the control circuitry126may be operative to perform the operations of one or more applications implemented on the electronics device120. Any suitable number or type of applications may be implemented. Although the following discussion will enumerate different applications, it will be understood that some or all of the applications may be combined into one or more applications. For example, the electronics device120may include an automatic speech recognition (ASR) application, a dialog application, a map application, a media application (e.g., QuickTime, MobileMusic.app, or MobileVideo.app), social networking applications (e.g., Facebook®, Twitter®, Etc.), an Internet browsing application, etc. In some embodiments, the electronics device120may include one or multiple applications operative to perform communications operations. For example, the electronics device120may include a messaging application, a mail application, a voicemail application, an instant messaging application (e.g., for chatting), a videoconferencing application, a fax application, or any other suitable application for performing any suitable communications operation.

In some embodiments, the electronics device120may include a microphone122. For example, electronics device120may include microphone122to allow the user to transmit audio (e.g., voice audio) for speech control and navigation of applications 1-N127, during a communications operation or as a means of establishing a communications operation or as an alternative to using a physical user interface. The microphone122may be incorporated in the electronics device120, or may be remotely coupled to the electronics device120. For example, the microphone122may be incorporated in wired headphones, the microphone122may be incorporated in a wireless headset, the microphone122may be incorporated in a remote control device, etc.

In one embodiment, the camera module128comprises one or more camera devices that include functionality for capturing still and video images, editing functionality, communication interoperability for sending, sharing, etc. photos/videos, etc.

In one embodiment, the visual app129comprises processes and/or programs for managing content, curating content and providing arrangement of display devices. The content may include: visual content, graphic images, video content, photos, etc. The visual app129may include any of the processing, curating, controlling, managing and arranging of content and/or display devices as described below.

In one embodiment, the electronics device120may include any other component suitable for performing a communications operation. For example, the electronics device120may include a power supply, ports, or interfaces for coupling to a host device, a secondary input mechanism (e.g., an ON/OFF switch), or any other suitable component.

FIGS.3A-Dshow an example of continuous lock of a virtual overlay410on the electronic device120for a display device140, according to an embodiment. In one example, as the user moves around holding the electronic device120, keeping the display device140within the field of view of the electronic device120, the virtual overlay is continually matched and locked to the respective display. In one embodiment, this is achieved by means of AR technologies that combine pattern matching (e.g., pattern matching using instance-level object detection, category-level object recognition, block-based visual-pattern matching, etc.), computer vision and computer graphics. In one example, a pattern matching processor (e.g., processor3111,FIG.29) or process (e.g., visual app129or content curation and display management processing3130), which is part of the AR library (e.g., part of visual app129and/or applications 1-N127), resides on electronic device120and is trained to detect a finite set of images. When the electronic device120is pointed at a display device140and if the image being displayed in the display device140is present in the electronic device120's pattern matching database, the pattern matching process or processor provides the AR library with details such as presence of an image, and the name of the image identified. The AR library then uses its computer vision process or processor to determine where in the scene this pattern is present, and details such as the location of its four corners. The AR library then uses its computer graphics process or processor to overlay desired visual effects within the detected four corners, that provide the illusion of a virtual overlay that remains locked to display device140.

In one example, the virtual overlay410may allow advanced user interfaces elements to be superimposed on top of them. These elements may include buttons and widgets that may be used to modify the state of the display device140(such as color, saturation, filters, etc.) to which the electronic device120is pointed at. Additionally, more advanced interactions such as dragging content/effects from a gallery on to the virtual overlay410are also possible. Putting all the above together, a user may detect and manipulate properties of a display device140simply by pointing at it with the electronic device120. This allows for very intuitive visual and spatially coherent interactions with display devices140and provides a satisfying user experience.

The above mentioned computer vision technology also provides the capability to measure the position, orientation, aspect ratio and dimensions of a display device140with respect to its neighbors. In one embodiment, the AR library simultaneously tracks multiple image targets using its pattern matching library (e.g., part of visual app129and/or applications 1-N127). This allows the electronic device120to setup these virtual overlays on top of each display device140in a cluster of displays which is displaying a familiar image. This collection of virtual overlays may then be analyzed spatially to determine which display device140is to the left of the other display devices140, above yet another display device140, etc. The relative positioning of a cluster of display devices140can thus be determined. Based on the relative orientations of the edges of one virtual overlay with respect to that of another, the electronic device120determines the relative orientations of these display devices140. The relative orientation of a cluster of display devices140can thus be determined. It should be noted that the aspect ratio of a display device140is the ratio between the width and height. Display devices140may have formatting in varying popular aspect ratios, such as 4:3, 16:9, 16:10 etc., depending on their use case and the appropriate media with the correct aspect ratio needs to be displayed in the display device140in order to avoid letterboxing or presence of black lines on the horizontal or vertical extremes. Each display device140inherently knows its own aspect ratio and physical dimensions. As soon as the electronic device120is pointed at display device140and identified, the electronic device120uses network communication to query display device140's aspect ratio and dimensions, which is communicated by display device140and used by electronic device120in further calculations. In one embodiment, the aspect ratio of display device140may be determined without network communication. With the virtual overlays mentioned above, electronic device120may compare the width and height of the virtual overlays to determine the observed aspect ratio of the display device140. When presented with a cluster of display devices140, this technology may be used to assess the relative placement, alignment and size differences of one display device140with respect to another display device140within the cluster. Since the above mentioned computer vision technology may be used to assess the placement, alignment and size of one display device140with respect to another display device140, it may be used to effectively spread a single image across multiple display devices140in a spatially coherent manner, that takes into account the gap between them (see, e.g.,FIGS.7D and8D).

In one embodiment, the multi-display device140environment illustrated inFIG.5Aprovides identification of physical aspects of the display devices140may be performed using the mobile device120, according to an embodiment. In one example, a single display device140or a cluster of display devices140, are also referred to as modular display devices140of the same or varying sizes. The display devices140are all connected on a same network. The network may be any wired or wireless communication channel such as Wi-Fi, BLUETOOTH®, LAN, WAN, cellular, etc. Each display device140has a unique network ID that may be used to assess and communicate with other devices connected to the network. Each display device140may or may not have touch or touchless gestural interaction features. In one example, an initial setup process may be required to add a display device140into a local network. For example, a user may pass local network configuration data from an electronic device120to each display device140through other technologies it may possess such as near field communication (NFC), USB, etc.

In one embodiment, the display devices140may include a spectrum of light emitting diodes (LEDs) (visual or infrared) or other physical markers on the bezel/border of the display device140frames, allowing distinct identification of each display device140using computer vision technology. In one example, a user has access to a mobile device120with a camera128, with input technologies such as multi-touch or air gesture, which is on the same network as the rest of the display devices140. The camera's128field of view may be adjustable to view a wide area or zoomed in to focus on a narrow area. In one example, the display devices140may also be connected via the network to other entertainment devices such as TVs and speakers.

In one embodiment, the electronic device120performs interactions with and commands a display device140by using the camera128to view the display device140and use of the visual app129. This interaction may be referred to as direct interaction/manipulation. In one example, tracking of display devices140is provided using the electronic device120, which is the ability to determine the position, orientation and size of each display device140with respect to other display devices140, the electronic device120and the room the display devices140are located.

Computer vision has progressed to a point where several processes may be used to track display devices140.Frame Markers: Display devices140may display specific predefined patterns known as markers, with which they may be identified. These markers typically take the form of checkerboard like patterns, that are unique. There are several types of these markers in varying complexities. In one example, QR codes and bar codes are specific examples of markers.Image targets: Advances in pattern matching provide use of regular images as ways to identify and track a display device140. This may be applied to either the currently displayed image, or each display device140may have a unique default image that it uses for the same purpose.Character recognition: a display device140can display specific text that may be used both for tracking and for visually communicating other meta-data.Watermarked markers: images displayed may be encrypted with invisible watermarks (which can only be picked up by the camera128) that may be used as a way to identify a display device140.LEDs embedded in the frame: visible/invisible LEDs or other light emitting elements may be embedded in the bezel of a display device140. These elements may then be sequenced to blink in a particular way and/or in specific colors, that will allow identifying and tracking the display device140.Physical features of frame: with modern AR software being able to pattern match 3D models and track in real time, display devices140with unique physical features may serve as a way for identifying and tracking a display device140.

One or more embodiments are described below with image targeting technology. However, it should be noted that other tracking techniques may also be used as well.

Communication link technology described herein uses computer vision technology at its core, which allows identifying a known image when viewed through a camera128of electronic device120. In one embodiment, all display devices140in the network can constantly communicate with each other to update each other with details such as name of the currently displayed image, their network ID, their aspect ratio, etc. As a result, every display device140on the network, including the electronic device120, contains a look up table that shows the relationship between network ID and currently displayed image. An example for a look up table provided below:

In one embodiment, when an electronic device120is pointed to a cluster of display devices140, it does not immediately know which display corresponds to which network ID. In one embodiment, the electronic device120has a pattern library, that constitutes its ‘known’ set of imagery. Each pattern is associated with an image identifier. The electronic device120used the visual app129to constantly scan each frame it captures with the camera128to see if any of the features in the captured image match with any content of the pattern library. When the electronic device120observes and recognizes a known image on a display120, it then uses the look up table to identify the network ID of that specific display device140and henceforth uses it to communicate any subsequent commands or changes to that display device140. The pattern library used by the electronic device120may reside on the electronic device120or be accessed on the cloud/server130(FIG.2). Every image captured via the camera128is put through a pre-processing stage, where the image is analyzed, defined as a pattern and added to the pattern library and to the cloud/server130. The cloud/server version of the pattern library may constantly synchronize with other users and other devices to ensure it is up to date with data. In one embodiment, all updates to the pattern library is done directly at the cloud/server. In another embodiment, device120or140initiates pattern library synchronization with the cloud/server every time it makes a change to its local pattern library. Each display device140may also be pre-loaded with a unique image, which serves as its default image and which in turn allows uniquely identifying the display device140.

In one embodiment, computer vision technology is also used to identify the physical dimensions of the display device140and its relative position and orientation with respect to the display121of the electronic device120. Computer graphics technologies may be employed to render digital/virtual overlays510(FIG.4B) on the display121of the electronic device120, that are on top of and encompass views of these display devices140.

FIG.4Ashows an example of a cluster of multiple display devices140, according to an embodiment. The relationship between the cluster of display devices140and their virtual overlays (e.g., overlays510,FIG.4B) may be captured into a snapshot referred to as a virtual wall, which is a combination of live imagery that shows the room/environment in which these display devices140are placed in (plants, desk, books, etc.) as well as their virtual overlays. One or more embodiments provide saving and retrieving of these virtual walls in order to manipulate them at any time. These virtual walls may be thought of as a photo of the room, captured by the camera128, where the content in the display devices140alone get updated with time. In one embodiment, the virtual walls may take the form of a standard rectangular image. In another embodiment, these virtual walls may be a panoramic image, such as a 360-degree (or other angle) image captured by either multiple cameras128on the electronic device120or sweeping around the electronic device120with a single camera128or other known methods. As the physical display devices140get updated with new images, the virtual walls containing these display devices140have their corresponding virtual overlay updated. Users may drag and drop content on to these virtual overlays within these virtual walls to update a real display device140, even when they are nowhere near the actual display device140. This allows for remote curation of a cluster of display devices140.

In one embodiment, once the user sets up a frame that satisfactorily covers all interested display devices140within its Field of View (FOV) and tracking has been initiated to draw the above mentioned overlays, several types of information can be obtained with this tracking data:a) The dimensions and aspect ratio of each display device140;b) The current orientation of each display device140(portrait or landscape);c) The relative sizes of display devices140with respect to each other;d) The position and orientation of display devices140with respect to each other (in other words, the gap between them);e) The distance of the electronic device120and hence the user from the each display device140and the plane/wall on which the display devices140are mounted; andf) The relative orientation of the user with respect to each display device140and the plane/wall on which the display devices140are mounted.
This rich list of information listed above provides the visual app129to add interesting capabilities and functionalities to display device140control, configuration and content curation.

In addition to tracking the display devices140, in one embodiment users may also supply ambient information through any of the following options:Stepping back or widening the FOV to include more of the ambient information in the room;Perform a separate pass, where the user captures just the ambience, taking a photo of the part of the room or object that they would like to be representative; andPerform a separate pass, where the user performs a movement/rotation to capture a stitched 360-degree image of the room, to be used for ambience calculations.

As seen inFIG.4A, three display devices140are hung on the wall. In this example, the three display devices140are in different shapes and sizes. It should be noted, however, that display devices140may assume the same shape or size, take various shapes or sizes, or additional display devices140or less display devices140may exist to form specific layout combinations. Each display device140displays content, which is either the factory default if the display device140has never been configured, or a user supplied content recorded by the pattern library through previous content management of the display device140.

In one embodiment, with the ambience information captured, it is possible to assess certain details of the ambience, such as color of the walls, lighting in the room, the color spectrum of all of the images shown in neighboring display devices140, etc. In one embodiment, the application running on electronic device120switches between two-dimension (2D) and AR mode. In one example, 2D mode allows for regular 2D interactions necessary for operations such as browsing image galleries and typing, etc. The AR mode loads the AR library and provides a camera view that is used to look at and interact with display devices140. In one embodiment, when the AR mode is turned on the electronic device120, the display devices140switch to a content that is conducive to one of the image based tracking technologies described above. In one embodiment, the displayed content may take the form of a pre-determined frame marker, watermarking etc., that is unique to that display device140. In another embodiment, the currently displayed image itself might serve as an image target and for tracking. The electronic device120is constantly communicated with the state of all display devices140and is correspondingly loaded with the right pattern recognition dataset.

FIG.4Bshows an electronic device120capturing a layout of the cluster of multiple display devices140ofFIG.4A, according to an embodiment. As shown, the electronic device120captures the layout of the display devices140ofFIG.4Aand provides an overlay510over each of the captured displays410. As the user points the camera128to the displays410, the electronic device120starts the pattern matching process described above to recognize each one of the display devices140.

In one embodiment, instead of solely relying on content recognition, the electronic device120is able to distinguish one display device140from another based on the LEDs embedded around the frame, edge or border of the display device140. For example, each display device140may have a unique LED lighting pattern. If there are X number of LEDs for each display device140, the unique pattern can reach X power of 2. For another example, the LEDs may increase the unique pattern by controlling the on/off frequency, on/off sequence, etc. In one embodiment, the electronic device120only needs to distinguish one display device140from another within the same group of captured display devices140. The same pattern can occur for another display device140as long as it belongs to a different group. This allows a smaller number of required unique patterns. The unique pattern may only need to present itself during a pairing process. In one embodiment, the pairing process includes loading the credentials for network communication such as Wi-Fi ID, BLUETOOTH® network etc., and its respective password onto electronic device120and transferring the same to a display device140via technologies such as NFC. Once display device140receives these credentials, it is then live on the wireless network, enabling wireless communication between electronic device120and display device140henceforth. Once the display device140is paired with the electronic device120, the identification of the display device140is associated with the corresponding virtual overlay as described above.

In one embodiment, when multiple display devices140are recognized, the visual app129identifies the region or pixels of each of the recognized display devices140in the image captured by the electronic device120. In one example, a box region (overlay510) is created for each recognized display device140corresponding to the identified region or pixels, and represents the corresponding virtual object. These box regions remain associated with the corresponding display device140, and each maintains the aspect ratio most suited for the display capability of the corresponding display device140.

FIGS.5A-Fshow a sequence diagram for an example of managing the image content displayed on the cluster of multiple display devices140, according to an embodiment. InFIGS.5A-F, three displays are shown, display640,641and642, all which are similar to display device140as described above. InFIGS.5A-5D, image content651is displayed in display641. In one example, the user can use the same control means as described above to upload content into desired display641. For example, by dragging a picture from an on-device album or gallery630to one of the box regions shown on the electronic device120.FIG.5Dshows dragging661from the album or gallery630to the box region associated with display641. InFIG.5Ethe image dragged from the album or gallery630is now displayed on display641as image content652, which replaced image content651.FIG.5Fshows the resulting wall of displays640,641and642.

FIGS.6A-Dshow a sequence diagram of an example for managing the image content711displayed on a display device701by adjusting scale, according to an embodiment. Users today have a large number of images in their collection. These images (e.g., photos, videos, etc.) are often taken with varying devices such as digital cameras, smartphones etc. In addition, they also receive/have access to images from various social platforms and services such as FACEBOOK®, INSTAGRAM®, PINTEREST®, SNAPCHAT®, etc. It would be expected that the user would like to share any of these images on their display701(similar to display device140as described above). Unfortunately, these images can come in various dimensions and aspect ratios and might not match 1:1 with the aspect ratio of the intended display701.

Auto cropping mechanisms such as center cropping the image on dropping to fit the display's701aspect ratio do not provide satisfactory results as region of interest in the image could be anywhere within the image and center cropping could cut out information or present them in an unflattering manner. Manually having to figure out the aspect ratio of the image, go into a third party app and perform cropping is tedious. Since the modular displays are always networked and constantly communicate with each other the details such as physical details and state, it is possible to know when the image being dropped on to a modular display will not match. In one embodiment, as shown inFIG.6Aon release of dragging an image content720from the album or gallery630, a new pop up730appears as shown inFIG.6B. The new pop up730shows the dragged image720along with a rectangle in the aspect ratio (the ratio of the width to the height) of the display that the user has dragged to. The user can then move and scale the box (with aspect ratio locked) until it best fits the regions the user would like to display. On confirming, this subset of the image alone will show up in the display701as the adjusted image content721as shown inFIG.7D.

In another embodiment, upon drag and drop and identifying a mismatch, the visual app129may also suggest another display device140in a group of displays that would be a better fit for this particular aspect ratio, so the user would not have to crop and compromise the look and feel of the image. This recommendation to use another display device140may also be based on other parameters such as colors of neighboring display devices140, motion of the content, resolution and other features. For example, consider a scenario where a cluster of display devices140are arranged on an environment (e.g., a wall), such that the display devices140on the left side show images that are predominantly on the red color spectrum and the display devices140on the right side show images that are predominantly on the blue color spectrum. An image is dragged and dropped that is predominantly orange in color, on to a display device140on the right side, the orange would appear glaringly out of place among the display devices140with predominantly blue images. One or more embodiments perform an analysis and suggest that the image might best be placed in one of the display devices140on the left for a visually pleasing effect. The user may then choose to select the recommendation or ignore it if they intended to have such a contrasting look (e.g., for creative reasons).

One or more embodiments provide for remotely managing the image content displayed on a display device (e.g., display device140), according to an embodiment. In one embodiment, a wall may include several display devices140. In one embodiment, when a user captures an image of the display devices on the wall with an electronic device120, the image also captures the contextual information surrounding the display devices, such as other wall decorations (e.g., surrounding furniture, color contrast among the fixtures, plants or other environment settings, etc.), which is then saved as a virtual wall. The contextual information available in these virtual wall images may improve the flexibility of user interaction with the modular devices. For example, the user may stay in her bedroom to manage the content of the modular devices in the living room. The virtual wall, which is a visual snapshot of a cluster of display devices140, can also be thought of as a list of network addresses such as an IP address embedded spatially within the snapshot. With the network address in hand, the respective display device140can be controlled from anywhere within the home. If the internal network is connected to external networks such as the Internet, the user has the ability to curate these display devices140from remote locations. Alternatively, another user that has access to the network address of the display devices140but is located remotely, can also curate the display devices140.

The user may use one or more virtual walls to manage the content display changes when she is on vacation, for example in the Caribbean (this example may require a home hub that can interact with the electronic device120so that the electronic device120can identify the specific modular display device to the hub, and through the hub to complete the content management). Alternatively, this can also be achieved via the cloud, if the multiple display devices are connected to the internet via technologies such as Wi-Fi.x1. The context information in these virtual walls allows her to not only view the content that will be displayed on each individual modular device, but also the overall effect of the content display in view of the surrounding setting. In this example, the user is able to save these virtual walls and categorize them with representative names such as hallway, guest room, living room, etc.

In addition to the above, users can share their virtual walls with friends, so they can briefly allow their friend to curate their photo wall. Alternatively, users can dedicate specific displays to certain friends, allowing them to curate that display. An example would be a mother dedicating a display to her daughter who is away at college. The daughter has access to this display and constantly updates it with her favorite images, to inform her mother of her life activities. This can provide for some fun and intimate interactions.

In one example, the image content is selected from an album or gallery. The user is able to drag and drop selected image content onto an overlay associated with a selected display device located on the wall. Upon release of the image content, the image content is now displayed on the selected display device.

FIGS.7A-Dshow diagrams of an example for managing the image content displayed on multiple display devices by spreading a single content across the multiple displays901,902and903(similar to display device140as described above) in a spatially coherent manner, according to an embodiment. In one example, image content is spread across the displays901,902and903in a spatially coherent manner.FIG.7Ashows a captured image of a layout of the displays901,902and903on an electronic device120with an album or gallery630available for content management and control. The displays901,902and903each show image content911,912and913, respectively. To spread an image across the displays, the user needs to perform 3 steps: a) first a menu option is selected, expressing intent to enter into this mode; b) the virtual overlays of displays901,902,903are then tapped to let the system know that it is across these chosen displays that it is desired to spread an image; c) an image may then be dragged and dropped onto the virtual overlays of any one of the selected displays. Each display is then sent a message to load the image of interest along with the texture coordinates valid for that particular display, which the display uses to ensure it displays only the portions of the image that it is supposed to. Texture coordinates are pairs of values used to indicate a specific sub-region of an image. The texture coordinates may be used to communicate the exact four points on an image of a display device901,902and903, the pixels within which a display device needs to render. Depending on the relative positions of each of the displays901,902and903with respect to each other and with respect to the electronic device120, their aspect ratios and physical dimensions, each display901,902and903will be calculated to subtend a particular sub-region of the image to be displayed and thereby the corresponding texture coordinates. The physical analogy would be to take a large print of a photo and then overlay it on top of a few picture frames and figure out which regions of the photo actually fall on top of which picture frames, cut them out, drop them into the respective picture frames, and discard the rest of the photo.

InFIG.7B, a user selects an image from the album or gallery630that she wants to be displayed across the displays901,902and903simultaneously, and drags930the selected image and drops it onto one of the displays901,902and903as shown inFIG.7C. InFIG.7Dthe dropped image gets spread across the three displays901,902and903as image content921,922and923. This technique enables a solution to maintain the spatial relationship of the image content over the displays901,902and903. It may be observed that the slope of the hill in the image content921,922and923is maintained. This would not have been the case if spatial relationship of frames had not been taken into consideration.

FIGS.8A-Dshow diagrams of an example for managing the image content displayed on multiple display devices by overlaying on top of a selected image content during the content selection process so that the image content is cropped over a whole group1000of display devices including the spaces among them, according to an embodiment.FIGS.8A-Dprovide an extended example of the concept shown inFIGS.7A-Dwith nine displays. In this example, instead of dividing the selected image content1030into nine parts according to the shape and size proportionally,FIG.8Cillustrates that the display devices overlay on top of the selected picture during the content selection process so that the image content is cropped over the whole group1000of the display devices, including the spaces among them. The user, thus, may understand how the image content will look like when it is displayed across the devices in the group1000. Note that each box region in the group of box regions1040ofFIG.8Ccontains a part of the image content1030. Moreover, all of the box regions altogether cover only a partial image of the selected image content1030.FIG.8Dillustrates how the selected image content is displayed by the multiple display devices with the display device1001showing its cropped image content1041. Note that the display devices display only the parts of the image that were overlapped in the corresponding box regions inFIG.8C.

In one embodiment, a user has the option to more flexibly adjust how the selected content will be displayed on each of the display devices. For example, via gestures such as slide (move up, move down, move left and move right), rotate, scale and crop, the selected image content1030changes its position, orientation, scale and boundaries relative to the group1040of the box regions (the relative positions of the box regions are fixed with respect to each other). In one example, the group1040of box regions remains still as the user manipulates the selected image content1030around those fixed box regions. The selected image content1030then populates into those fixed box regions as desired. This allows the user to fit the most interesting content in the best way possible, to their liking.

FIGS.9A-Fshow diagrams of an example for managing the image content displayed on display devices1101,1102and1103by spreading a single content across these multiple display devices based on regions of interest, according to an embodiment. In one example, a technique for spreading a single content across multiple display devices1101,1102and1103is based on the box regions (of interest)1131,1132and1133.FIG.9Ashows the display devices1101,1102and1103secured onto the wall and a captured image of them on an electronic device120. The display devices1101,1102and1103are currently showing image content1111,1112and1113, respectively.

The user selects image content from an album or gallery630that she wants to be displayed across these display devices1101,1102and1103simultaneously and drags and drops it into any one of those displays shown on the electronic device120.FIG.9Billustrates an initial view of how the selected image will be displayed across the display devices1101,1102and1103as image content1121,1122and1123, based on the original box regions1131,1132and1133. As illustrated byFIGS.9B-F, only the part of the selected image that falls within a box region will be displayed.

InFIG.9B, the body of the people in the selected image content are not correctly/fully included in any of the box regions1131,1132and1133. One or more embodiments provide the ability to upload the most desired portion of a selected image content into any of the display devices. Each box region is designated to a particular display and represents what pixels are drawn on that particular display. The aspect ratio of a box region is always locked to that of the display device it is assigned to (even when its scale changes, it maintains the aspect ratio).FIG.9Cshows that a user is able to select any of the box regions1131,1132and1133and drag one of its corners1140to change the area that this box region covers.

FIGS.9D-Eshow how a box region can be moved so that it overlays the desired portion of the selected image content. As described above, a box region on the captured image is associated with a particular display device. When the box region has its corners1141dragged, moved or scaled, the process tracks the changes and records the new position of the box region without losing the display device association.FIG.9Fshows the final result of such operation that each display device1101,1102and1103is able to display the exact portions1121,1122and1123of the selected image content, as designated by the user via the electronic device120.

FIGS.10A-Dshow diagrams of an example for managing the image content displayed on display devices by changing the relative relationship of the image content1211displayed on a display device1201, according to one embodiment. In one example, the visual app129provides the ability for a user to change the relative relationship of the image content displayed on the display devices. In one embodiment, the user is able to move a box region corresponding to a display device on the right side to overlay on any part of the selected image1240(selected from the album or gallery1230inFIG.10B). For example, the display device1201on the right may display an image of the kid on the right side of the selected image1240. The user is also able to scale the size of the box region1250to adjust the portion of the image content that will be actually displayed on the corresponding display device1201. This allows the user to change the ratio of different aspects of the image content displayed by the display devices, with respect to the overall selected image1240. In another embodiment, a small inset image is shown that showcases the original alignment of each box with respect to the image content that serves as a reference for the user. Once the box region1250is dragged to the portion of the selected image content1240as desired (and adjusted for scaling), the final image content1221is displayed on the display device1201inFIG.10D.

FIGS.11A-Fshow a sequence diagram of an example for sharing image content shown in any of the multiple display devices, according to one embodiment.FIG.11Ashows eight display devices (similar to display device140). InFIG.11Ba captured image of the display devices (including display device1301showing image content1311) is shown on the electronic device120, along with an album or gallery1330. Similarly as a user may drag and drop an image content from the electronic device120on to a display device with an overlay, they may do the opposite as well in order to save or share the image content. In one example, a user may point their electronic device120to a display device and obtain access to the original image content (e.g., image content1311on display device1301) displayed in the display device for purposes such as saving, sharing and swapping.

Typically, if a user would like to share an image/video that is currently displayed in one of the display devices with a friend, they would need to navigate through the plethora (e.g., 100 s, 1000 s) of photos in their mobile device to search, identify, attach to a message or email and share it with another user. In one embodiment, using an AR technique, the visual app129identifies the photo/video, links directly to the local copy stored in its gallery or album and makes it available immediately for further operations, such as sharing with a friend, sharing with another mobile app, etc. The AR tracking here works analogous to shortcuts on desktop computers.

The network lookup table described above along with virtual overlays is useful in identifying a display based on its currently displayed image. The same look up table may also be used to obtain the original image that a particular display device140is currently displaying. When the user directly touches a virtual overlay associated with a display device140, the system then uses the network address associated with the display on the look up table to retrieve the path to the image that the display is currently showing. Once this path is retrieved, the system can proceed to do any number of operations with it. InFIG.11Bthe box region1320is selected by a user. InFIG.11Cthe selected image content1311in the box region1320is dragged to the share portion1340of the display on the electronic device120as shown inFIG.11D. InFIG.11E, a selection graphic1350(e.g., a window, a pop-up, etc.) is displayed with the selected image content1311providing options for sharing, forwarding, saving, etc. (e.g., via text message, email, messenger, saving to a gallery or album, social media platform, etc.). In one example, the user selects to share the image content1311in a text message, andFIG.11Fshows a text message display1360with the image content1311that may be forwarded to others as desired. In another example, the user may select the image content1311in the box region1320and drag the image content1311to a portion of the display on the electronic device120such as the gallery or album, a favorites selection, etc. for saving the image content1311as desired.

FIGS.12A-Bshow an example illustration of finger movement drag and drop for moving image content between display devices1401,1402and1403, according to one embodiment. In one example, display devices1401,1402and1403are displaying image content1411,1412and1413, respectively. In this example, a user desires to move the image content1412in the display device1402to the display device1401. The user may grab the image content in the box region1440shown on the electronic device120by dragging and dropping it on to the desired display, in this example the box region associated with display device1401. This way, the user may swap and reorganize the same set of images among the cluster of display devices, without having to reach into their gallery or album1430. The result is shown inFIG.12Bwhere the content image1412is shown in display device1401. In one example, another image content1422takes the place of the moved image content1412. The image content1422may be a default image, a next image in the gallery or album1430, a previously displayed image content (e.g., the last displayed image content or a shuffle selection) recorded by the display device, etc. In another embodiment, the content1422might be the same as1412i.e., a simple swap of images happens between displays1401and1402.

FIGS.13A-Bshow an example illustration of finger movement drag and drop for applying predetermined filters or filters based on ambient colors for display devices, according to one embodiment. In addition to dragging and dropping image content on to a display device, the same technique may be used to apply effects on a particular display device.FIG.13Ashows an example, where the user points the electronic device120running the visual app129at the display device1501that is displaying image content1511. Once the display device1501is recognized, the user can select an option from a list of image effects1530such as ‘black and white’, ‘sepia’, ‘vignette’, etc. and drop them on to the display device show in on the electronic device120. The command is conveyed via the network and the intended display device applies the selected effect1531as updated image content1521shown on display device1501inFIG.13B.

In one embodiment, the electronic device120also detects the color of the wall that the display device(s) (e.g., display device1501) is(are) hung/attached to and the lighting color in the room—referred to as ambient colors. Considering the color scheme of the content image(s) showcased in the display device(s), and the color scheme in the ambience, the visual app129provides a recommendation for color filters, that can neutralize the overall color palette of the collection and allows them to blend into the room in a more visually pleasing manner.

FIGS.14A-Cshow an example illustration of selecting colors from the image content1611being displayed to be used as the mat color for display devices (e.g., display device1601), according to one embodiment. Picture frames available on the market typically are provided with a mat. Mats usually serve the purpose of bringing attention to the displayed image content by providing a color relief and separation from the plethora of information surrounding the frame. These mats are mostly made of paper, but do also come in a variety of textures and colors.FIG.14Ashows a display device1601having a mat portion1620and image content1611. In one example, it is desired to modify the mat portion1620. InFIG.14Bthe captured image content is shown on the electronic device120. A graphic or icon1630is used to select a portion of the image content shown on the electronic device120. InFIG.14Cthe graphic or icon1630is dragged to a portion of the image content and a color representation (e.g., a graphic display of the color selected)1640of the color of the portion of the image content where the graphic or icon1630is placed is displayed. The image mat1620is then changed to mat1621to match the selected representation1640and displayed on the display device1601.

FIG.15shows an example illustration of grouping of display devices in multiple areas, according to one embodiment. The home of today may be filled with display devices (e.g., display devices140) of varying shapes and sizes—television sets, digital picture frames, tablets, etc. These display devices are also typically set in different parts of the home. Especially when it comes to digital picture frames, people may prefer to decorate different regions of the home with images of different themes, albums, etc. In some cases, people also break down a cluster of picture frames into two or more themes. For example, image content on the wall1700in one room (e.g., a living room) in display devices1701and1702, and image content on the right on wall1710in display devices1703,1704and705in another room (e.g., a kitchen). Other scenarios such as certain image content on the left, center and right of a wall may have separate themes of image content. For example, boy's pictures on the left of a wall, family or parent pictures in the middle of the wall, and girl's pictures on the right of a wall. In such situations, there arises a need to group a set of display device.

In one embodiment, grouping display devices provides the following benefits:Provides interacting/ modifying with a group of display devices without disturbing another neighboring (or far away) group;Provides setting characteristics that apply to all display devices in a group—such as group effects/filters, slide show animation speed, etc.; andProvides curation of all display devices in a group with one quick interaction.

In the example shown inFIG.15, the display devices1701and1702are set up as a group1711(e.g., a color or other indication may be associated with the group for easy visual recognition when captured on an electronic device120(FIG.2)), and the display devices1703,1704and1705are set up as a group1712(which would be another color or other indication to provide recognition). This clean distinction prevents any interaction on the living room (wall1700) images to adversely affect the set of display devices in the kitchen (wall1710) and vice versa.

FIGS.16A-Cshow an example illustration of managing groups of display devices using an electronic device120, according to one embodiment. There are several interaction opportunities for both the creation of display device groups and for modifying/curating them. In one example, users may use interactions to create the desired groups.FIG.16Ashows an example where a tray of indicators (e.g., colors)1810and1815shows up in the electronic device120. The display devices1801,1802,1803,1804,1805and1806are desired to be placed into different groupings of1810and1815. In one example, the group1810includes display devices1801,1802,1803and1804. The group1815includes display devices1805and1806. Users can select the grouping selection from selections1830, point their electronic device120at a display device and drag and drop an indicator (e.g., a color) on to the represented or captured display device. On the electronic device120, indicators (e.g., colors) for the two groups (1810and1815) are shown on the bottom portion of the electronic device120, along with a selection for adding more groups.

InFIG.16Ba selection is made for changing display device1801from group1810to group1815using drag and dropping of the group1815indicator onto the representative image1850for display device1801.FIG.16Cshows completion of the assignment of display device1801to group1815. Upon completion, the display device1801is now assigned to group1815(e.g., an orange group). All displays assigned with the group1815indicator (e.g., orange color) will become part of the same group. Similarly, displays assigned with a group1810indicator (e.g., blue color) will all become part of the blue group. Users can select the plus (+) button on the left to create more such groups. Users may dynamically arrange and rearrange groups as they please. Any display device can at any time be made to:Join a group;Leave a group and join another group; andBecome its own island group, with just that display device within that group.

Group curation actions may in general be classified into four subsets: One-to-one; One-to-many; Many-to-many; and Many-to-one. The prefix denotes the nature of source selection on the electronic device120, and the suffix denotes the nature of destination assignment on drop. The behavior and output of actions change based on which of the above modes that the user chooses.

One-to-one refers to the technique where a user may perform one action to influence one display device. For example, the drag and drop technique is used to apply a single image content to a single display device as described above is an example of a one-to-one interaction. Similarly, dragging and dropping a filter to apply to a single display would be one-to-one.

One-to-many refers to a technique used when a set of display devices are grouped. Users may perform familiar drag and drop actions to influence one action on all the display devices in a group. For example, to apply a filter effect to every display device in a group, a user may drag and drop the filter on to any display device in the group and the rest of the display devices in the same group will also be applied with the same filter.

Many-to-many refers to the technique when instead of selecting a single image content from an album and dragging it, if the user drags the album icon and drops it on to any display device in the group, all display devices in the group are automatically loaded with a random image from that album.

Many-to-one refers to actions, where multiple content/actions are assigned to a single display device. For example, users could drag and drop an album icon on to a single display device to influence just that display device, such that all images in the dragged album form a stack and the display device iterates through them with a slideshow.

In one embodiment, there is a clear need for ways in which users may express their intent to select/switch to any of those four grouping modes before/during their interaction to obtain desired effect/output. In one example, the user may in advance select the grouping mode for consecutive action to be made in one-to-one mode. Here the user selects, for example, a toggle button to switch to ‘Single’ mode and drags and drops a filter. As the user drags and hovers a black and white filter over a display in a blue group, only that particular display device highlights to show that it will be affected by this action. On drop, only that particular display device gets applied with a black and white filter.

In another example, a situation may arise where the user in advance selects the grouping mode for a consecutive action to be made in one-to-many mode. Here the user selects the toggle button to switch to ‘Group’ mode and drags and drops a filter. As the user drags and hovers over a black and white filter over a display in the blue group, all displays in the blue group highlight to show who will be affected by this action. On drop, all displays in blue group get applied with a black and white filter.

In yet another example, a user selects the grouping mode for consecutive action to be made during the drop interaction. As the user drags and hovers a black and white filter over a display device, the overlay splits into two halves/sub-menus, providing options between ‘single’ and ‘group’ mode. If the user drags and hovers over the ‘group’ sub-menu, all other display devices in the same group (e.g., a blue group), highlight to show which would be affected by this action. On the other hand, if the user drags and hovers to ‘single’ sub-menu, only that display device becomes highlighted. Depending on which of these two sub-menus the user drops the dragged filter to, the black and white filter is applied as one-to-one or one-to-many. Alternatively, the user may also select the grouping mode for consecutive action as they begin the drag operation. For example, as the user starts to drag a black and white filter, two sub-menus appear near the point of drag. One represents a ‘group’ action and the other represents a ‘single’ action. If the user drags the effect to either sub-menu, that particular sub-menu gets activated as the mode during drop. InFIG.20A, the user drags and makes a first stop at single mode sub-menu, activating one-to-one curation, and then continues on to hover and drop the effect on a display, just affecting that display. If the user drags and makes a first stop at a group mode sub-menu, activating one-to-many curation, and then continues on to hover and drop the effect on a display device from the blue group, affecting all displays in the blue group.

There are several models of behavior possible within a group:Albums may dictate group indications (e.g., color), that is there is a strict correlation between indication and albums. For example, all display devices showing pictures from a Christmas album have to be part of the red group. When a display device/or the group of display devices is assigned the beach album, they have to switch to becoming a blue group (e.g., the beach album is associated with color blue);Display devices may be part of the same group, but do not necessarily have to show image content from the same album; andFollow everyone else in the group. For example, groups may retain their color and switch between albums, but all display devices in the group have to switch to the new album as well (i.e., all display device show the same album).

FIGS.17A-Bshow an example illustration of an example template recommendation that measures the display devices and provides recommendations of various configurations for multiple display devices, according to one embodiment. A ledge, which is a horizontal bar affixed to the wall is used to mount the displays. Electronic device120already knows the details of the ledge, such as its dimensions, spacing of mounting slots etc., and when switched to ledge mode, takes these details into consideration for its calculations. As illustrated,FIG.17Ashows display devices1901,1902,1903,1904,1905,1906,1907and1908placed on shelves on a wall. In one example, in a ledge mode, the display devices may be found to not necessarily align well with each other. In one example, the electronic device120captures an image of the overall ledge layout. Each display device1901,1902,1903,1904,1905,1906,1907in the ledge mode is recognized through pattern matching, and is paired with the electronic device120. In one embodiment, the electronic device120obtains the length and width of each display device during the pairing process. In one example, the length and width of each display may be gathered through a separate process. In one example, the pattern matching recognizes three patterns for the display devices1901,1902,1903,1904,1905,1906,1907as patterns1920,1921and1922as shown inFIG.17A.

As the size of the ledge is fixed and known to the electronic device120, with the knowledge of the total number of display devices, and the size and shape of each, the electronic device120is able to make recommendations of how to place the display devices in the ledge mode to the user.

Each assembly manner is referred to as a template. As illustrated inFIG.17B, every template shows several numbers. Inside the frame of the ledge, there is marked measurement1910that the user can place (unless pre-installed) and follow the numbers provided in the template to place the display devices. As shown, the templates may be selected from a template selection portion shown on the electronic device120. InFIG.17B, the template1930is selected and the outline of the display devices based on shape are displayed in display1940along with the numbers representing positions or distance/measurement according to the marked measurement1910. The user simply places the display devices based on size/shape at the number on the marked measurement on each ledge according to the template. This assists the user in arranging the display devices (e.g., display devices1901,1902,1903,1904,1905,1906,1907) on the ledge or shelf.

FIGS.18A-Bshow an example illustration of an example template recommendation for an overall canvas layout2010for multiple display devices, according to one embodiment. A canvas, which is a flat panel affixed to the wall, is used to mount the displays. Electronic device120already knows the details of the canvas, such as its dimensions, the spacing of mounting slots etc., and when switched to canvas mode, takes these details into consideration for its calculations. In one example, a picture is captured of the overall canvas2010layout using electronic device120. The size of the canvas2010is fixed and known to the electronic device120. The visual app129(FIG.2) may calculate the number of possible combinations of placement of the display devices based on the size of the canvas2010, the sizes and shapes of the available display devices (e.g., display devices140,FIG.2), the power charging positions of such display devices, the power charging positions available on the canvas2010, etc. (this information may be retrieved during the pairing process or through a separate process, for example, discovery of the canvas2010).

In canvas mode, the template recommendation is not limited to align a display device along the horizontal direction as in the ledge mode, but also along the vertical direction. Each template2030may also indicate the position marks on a display2040that match the same marks on the canvas2010to help a user fasten the display devices following the template recommendation. With an easy tool such as this, users can easily set up a wall of display devices in a few hours with complicated arrangements such as that in a heart shape, crescent moon shape, etc., which would have traditionally taken days of planning, measuring, aligning, etc.

In one embodiment, template recommendations may be applied to a tile mode as well. In one embodiment, the electronic device120captures an image of the layout of the tiles using her smart phone. The user is allowed to carve out certain areas or tiles from a picture indicating that display devices should not be placed at these positions. The visual app129(FIG.2) then recommends where to place display devices for the remaining available tiles, for the selected content, to achieve the most aesthetic display effect. In situations where the ledge or canvas used by user is unknown or when the user mounts on surfaces such as tabletops whose dimensions are unknown, the system is able to provide options and recommendations for possible layouts, but would not be able to provide specific measurements or positions to where the display devices140would need to be placed.

Arrangement recommendations can also be influenced by other factors. Because these display devices are networked, it is possible to then bring out a social element. In one example, real-time information as to what are the popular arrangement styles that users across the country prefer may be collected and recommended to others. On the contrary, if someone prefers a unique arrangement, recommendations for the least chosen arrangements may be made. Certain arrangements are geometrically symmetric and are more visually pleasing. These may be made as special recommendations for the user. In one embodiment, the arrangements a user had previously used may be tracked, such that these arrangements may be switched back to rather easily. This allows for temporary re-arrangement of displays during parties, holiday season, etc., but the arrangements may then be placed back exactly how they were before the re-arrangements. In one embodiment, the template recommendations are made based on the content selected to be displayed in the cluster of displays. For example, in special scenarios such as a single image is spread across multiple images randomly, random template arrangements, may affect the aesthetic feel of the image being displayed. The system at this point can recognize that a single image is being spread across multiple displays and only suggest valid template options or recommendations for the involved displays.

In one embodiment, in any of the above mentioned arrangement modes, the mounting surface (ledge/canvas) may contain LED lights distributed across the mount. Once calculations are done by the electronic device120(or separately processed), these LED lights may blink, so the user can directly place the display device at that location. This removes the necessity to even look for numbers or a measurement. In another embodiment, when these mounting surfaces contain sensors to assess placement/mounting of display devices, they may also confirm that the user has performed the right operation. The electronic device120may then notify the user at every stage of accomplishment in performing the mounting/arranging operation, guiding them. This significantly reduces the cognitive load of the user and allows them to enjoy the act of mounting and arranging display devices.

FIG.19shows an example illustration of different viewing perspectives for multiple display devices on a wall2110. As shown, a person2130standing closer to the wall2110has a different viewing perspective than a person2140standing farther away from the wall2110. On the right side ofFIG.19, the perspectives of the person2130and2140are represented by point2131and2141, respectively, in a geometrical graphical representation2120. The difference in perspectives affects the way the wall2110of display devices may be perceived.

FIGS.20A-Dshow an example illustration of determining the position of an ideal view point, based on the size, shape and overall layout of multiple display devices, according to one embodiment. As shown inFIG.20A, multiple display devices (e.g., display devices140,FIG.2) are arranged on a wall2210. InFIG.20B, an electronic device120captures a view of the wall2210. InFIG.20Cthe electronic device120performs measurements to guide users to adjust arrangement of the display devices, and the electronic device120shows a progress display2220as measuring. When users are securing the display devices, for example, onto wall2210, they are close to the display devices such that they cannot have a good view of the overall effect from the usual viewing position of a distance away from the wall2210. Additionally, being that close provides a very distorted perspective view, which makes it hard to read straight lines and hence estimate how well a display device is aligned with the horizontal, or with respect to another display device. Typically, the ideal spot that provides the best and least distorted viewing experience is some distance away from the wall2210and directly facing the wall2210(e.g., the person2140,FIG.19as compared to the person2130). InFIG.20D, after taking measurements and processing the display devices on the wall2210, a projected view2230is provided, which is a generated image of a view of the wall2210from the ideal viewing spot or position of the display devices with content displayed.

In one embodiment, the user that is arranging the display devices may take a picture or a video of all the display devices from any perspective, using the electronic device120. A perfectly rectangular object, for example, may appear deformed when the image is taken from a perspective that is not perpendicular to its plane (e.g., the image on the electronic device shown inFIG.20B). Once the display devices are paired with the electronic device120to obtain required information such as the image displayed in each electronic device120, their physical dimensions etc., the sensor(s) of the electronic device120such as a camera may further determine the distance and angle of the electronic device120to the display devices. Alternatively, as the sizes and shapes of the display devices are known to the electronic device120, the degree of the deformation may be used to calculate the relative position of the user, with respect to the deformed object in the image. As the user moves closer and further away from a display device140, the size of its virtual overlay increases and decreases respectively. When the user is exactly in front of the display device140, its ideal viewpoint, the virtual overlay associated with display device140has edges that are perfectly at 90 degrees to each other. The more the user moves away from the normal axis sideways or top down, the edges of the virtual overlays stray more from the 90-degree angle to match the distorted perspective. This information may be mathematically generalized using principles of projective geometry, homography, etc., and compared with the known physical dimensions of the display to estimate exactly where, with respect to display device140, that the user and the electronic device120is currently positioned. Once the current position and viewpoint is calculated, the amount of counter rotation and movement required to obtain the ideal viewpoint is then calculated.

In one example, the electronic device120is able to determine the position of an ideal view point, based on the size, shape and overall layout of the display devices. The ideal view point would be a position, and at a reasonable distance from the wall, that the electronic device120can take an image without causing any deformation. In one embodiment, the visual app129(FIG.2) may calculate how to counter rotate the perspective and crop the captured image by virtually moving the electronic device120from the real position recognized to the ideal view point. The orientation of the image may thus be adjusted as if the image is captured by the electronic device120at the ideal view point. This way, the user can follow the cropped image to adjust the arrangement of the display devices without requiring additional help from another user, or constantly moving to the ideal viewing spot to check on the display effect. In one embodiment, the electronic device120allows the viewer to view not just from this ideal viewpoint, but from any angle and distance with respect to the display devices, as permissible by the resolution of the image. For example, the user can identify a preferred viewpoint, which is not the ideal view point. The cropped image will allow the user to see the effect of the arrangement of the display devices from the preferred viewpoint. In one embodiment, the preferred viewpoint may be determined by visual app129, based on user preference learned from previous arrangements, and/or environmental data surrounding the display devices (e.g., size of the display space, furniture and/or plants around the display devices that may block the view of the display devices, areas that have the line sight to the display devices that the user frequently presides, etc.).

FIGS.21A-Dshow an example illustration of capturing a frame, analyzing the prominent colors in a room2310and displaying these colors as a scrollable list2320and presenting images2330/2331based on the list2320on an electronic device120, according to one embodiment. In one example, the detected ambient colors of a room2310may be used to recommend images/art work from either the user's collection (e.g., gallery, albums, etc.) or from curated services, that best match the color palette of the ambience in the room2310. In one example,FIG.21Ashows a user pointing the electronic device120camera128(FIG.2) at their room2310. The electronic device120captures a frame, analyzes the prominent colors in the room2310and displays them as a scrollable list2320at the bottom of the screen. When the user presses on any of the colors (e.g., color2321) in the list2320, the visual app129then sifts through its database, looking for images which have this selected color as its dominant color and presents them to the user as a scrollable collection of images2330as shown inFIG.21C. Similarly, a selection of color2321results in a collection of images2331inFIG.21C. The user can select an image of interest and use this for curation of images for the display devices on the wall2310.

FIGS.22A-Cshow an example illustration of assessing the color of a wall2410surrounding a display device frame and using that information to color match the mat2420of the display device, according to one embodiment. Most picture frames available on the market come with a mat. Mats usually serve the purpose of bringing attention to the picture by providing color relief and separation from the plethora of information surrounding the frame. These mats are mostly made of paper, but do also come in a variety of textures and colors. Some picture frames have transparent mats, which are see through and pick up the color of the wall behind it. With digital display devices, one embodiment simulates the experience of a mat using digital mats, which serve the same purpose as their analog picture frame counter parts.

In one embodiment, a computer vision technique is used to assess the color of the wall2410surrounding a display device and using that information to color the mat2420. In one example, the electronic device120captures the image of the wall2410, analyzes the color components of the wall2410, and provides a selection of mats2430matching the wall2410color(s) inFIG.22B. InFIG.22C, a selection of a mat from the selection of mats2430is made and the mat2420of the center display device is changed to mat2421.

FIGS.23A-Cshow an example illustration of identifying repeating patterns on a wall2510, and digitally synthesizing the patterns to provide a similar transparent mat effect for a display device, according to one embodiment. In one example, an effect achieved is similar to that of having a transparent mat (e.g., clear or see through glass) around photo/video content on the display device of the wall2510. Walls with patterned designs such as tiles, bricks and wallpaper are popular in many parts of the world. One embodiment may be extended to these patterned walls. If the repeating patterns are small enough, a single image is captured that contains at least a few examples of the pattern, fully visible and unoccluded by other objects. The system then identifies this repeating pattern and stores it for future processing. If the repeating patterns are too large, or the room is setup in a manner that the fully visible examples are far from each other, the user can achieve a similar effect by stitching together multiple overlapping images to form a single large image. A step is performed, where multiple images of the wall2510on which the display device is mounted. These images are then processed using computer vision and stitched together to form one large image. In one embodiment, if the electronic device supports it, a single large image covering the display device (and any other display devices) and a substantial portion of the wall2510will also do. A process analyzes the stitched image to identify repeating patterns, their arrangement and spatial relationship to the location of the display device (e.g., pattern matching using instance-level object detection, category-level object recognition, block-based visual-pattern matching, etc.). In one embodiment, the system runs a process that performs the following: a) the image is analyzed to identify a pattern that is clearly repeating across the image; b) the identified pattern is then separated via cropping and stored in a database for reference along with other meta data (referred to herein as the reference pattern); c) the immediate borders surrounding each display device140is then analyzed for presence of patterns; d) these border patterns are then checked to see if they are a subset of the identified reference pattern; e) once there is a match, then it is determined that the detected border pattern is actually larger, consisting of all of the information as that of the reference pattern, and portions of it being occluded by the display device140; f) subsequently, the regions of the pattern that would have been visible but are currently occluded by display device140's mat2520area are digitally synthesized via region filling and rendered in the accurate location.FIG.23Cshows the result mat2521matching the pattern on the wall2510to provide a transparent mat effect.

In another example, a wall may have a brick texture pattern. In this example, a display device on the wall originally has a white mat. In this example, the user desires to have a clear mat effect on the display device. An electronic device120captures an image of the wall and analyzes the wall pattern to simulate a clear mat such that the pattern of the wall continues in the mat region of the display device. The mat is changed to a mat that appears to be clear as the pattern of the wall continues into the mat region.

When the user has the ability to capture an image of the wall without the display devices140, either prior to mounting or is able to temporarily remove and mount the display devices140, another example is described below. The underlying technology may be thought of as a subtraction mechanism. In one embodiment, an image A of the wall is captured without the display devices140. Then the display devices140and the respective images are placed at their correct locations and another image B is captured. Computer vision technology is used to align the two images A and B, and a one-on-one comparison is processed to figure out regions C that are visible in image A, but are being occluded in image B. The respective pixels from C that lie on the mat area of each of these display devices140are then determined. Texture coordinates for these regions are calculated and then sent to the respective display devices, which then display content from original image A in the mat area to provide an illusion of a clear mat. This embodiment is especially useful in scenarios when every instance of the repeating pattern is occluded in one way or the other, making it hard for other embodiment processing to execute. This technique is also very valuable in scenarios where instead of a repeating pattern, the wall is decorated with one large design such as a photo, painting or mural where the above embodiment is not usable due to absence of a repeating pattern.

FIGS.24A-Cshow an example of customizing matting for imaging on a display device, according to an embodiment. In one embodiment, control is provided to the user as to how much of a mat they would like to see. In the example shown inFIG.24A, the electronic device120camera128(FIG.2) is pointed at a display device2610having a mat2615. An outer mat control interface2620and an inner mat control interface2630are displayed on the electronic device120. In the example shown inFIG.24B, the inner mat control interface2630is used to adjust the thickness of the mat. In this example, the mat2615thickness between the four images is increased, resulting in new mat2616. In the example shown inFIG.24C, the outer mat control interface2620is used to increase the thickness of the mat2616, resulting in the new mat2617. It should be noted that the mat thickness may be decreased as well if desired by adjusting the scroll bars for the outer mat control interface2620and the inner mat control interface2630.

FIGS.25A-Bshows another example of customizing matting for imaging on a display device, according to an embodiment.FIG.25Ashows an example where the electronic device120camera128(FIG.2) is pointed at a display device2710having a mat2715. As shown inFIG.25B, the mat may be touched and dragged from the corner2720either inwards or outwards to increase, decrease or remove altogether the mat2715. In this example, the mat2715has its thickness increased resulting in mat2716shown on the display2710.

FIG.26shows a block diagram of a process2800for overlaying visual effects on a captured image, according to one embodiment. In one embodiment, in block2810process2800provides, when the electronic device is pointed at a display device (e.g., a display device140), capturing of an image displayed in the display device on an electronic device (e.g., with a camera128of electronic device120,FIG.2). In block2820, a pattern of the image is matched in a pattern matching database (e.g., using a matching processor (e.g., processor3111,FIG.29) or process (e.g., visual app129,FIG.2or content curation and display management processing3130), which is part of the AR library (e.g., part of visual app129and/or applications 1-N127), resides on electronic device120and is trained to detect a finite set of images). In block2830, if the image is present in the pattern matching database, the details are provided to an AR library, such as presence of an image, the name of the image identified, etc. In block2840process2800detects where in a scene the pattern is present and the location of the four corners of the pattern. In one example, the AR library uses its computer vision process or processor to determine where in the scene this pattern is present and the details. In block2850, visual effects are overlayed within the detected four corners for providing a virtual overlay that remains locked to a display device of the electronic device. In one example, the AR library uses its computer graphics process or processor to overlay desired visual effects within the detected four corners, that provide the illusion of a virtual overlay that remains locked to the display device.

FIG.27shows a block diagram of a process2900for spreading an image over multiple display devices, according to one embodiment. In one embodiment, block2910provides determining relative positions of each of multiple display devices (e.g., display devices901,902and903,FIGS.7A-7D) with respect to each other and with respect to an electronic device (e.g., electronic device120). In block2920process2900provides determining aspect ratios and physical dimensions for each of the multiple display devices to subtend a particular sub-region of an image to display. In block2930process2900provides for associating corresponding texture coordinates for each of the multiple display devices. The texture coordinates indicating a portion of the image to display on each of the multiple display devices. In block2940process2900provides for sending each of the multiple display devices a message including the image and associated texture coordinates. In block2950process2900provides causing loading of the image across the multiple display devices based on the associated texture coordinates.

FIG.28shows a block diagram of a process3000for configuring and controlling display devices, according to one embodiment. In one embodiment, in block3010process3000receives an image of one or more display devices (e.g., display devices140,FIG.2) with respective first content displayed. In block3020process3000retrieves, for at least one display device, a display device ID. In block3030, a change to the first content displayed at the at least one display device is caused, based on the display device ID of the at least one display device, upon interaction with the image.

In one embodiment, process3000may further include delineating regions, on the image, via a computer vision technique, where each region corresponds to a distinctive display device. Interaction with the image including interaction with at least one of the regions. Process3000may additionally cause a second content to be sent to at least two display devices for display. The second content is displayed across the at least two display devices maintaining the spatial relationship of the second content.

In one embodiment, process3000may further include performing content recognition based on display device ID using computer vision. Additionally, one or more display devices may be linked to a network and content recognition may be performed accessing a pattern library. In one example, a user interface (e.g., using electronic device120,FIG.2) provides selection of display devices and interaction with the selected display devices.

In one embodiment, process3000may include overlaying one or more display devices on a user interface to enable various gestures for managing content displayed on the one or more display devices. Additionally, process3000may include receiving a selection of a portion of an image of the first content, receiving instructions to crop the portion, and causing the cropped portion to be displayed in a selected display device of the one or more display devices.

In one embodiment, process3000may include receiving a selection of one or more portions of an image of the second content, receiving cropping instructions to crop the one or more portions, and causing the cropped one or more portions to be displayed across the at least two display devices.

FIG.29is a high-level block diagram showing an information processing system comprising a computing system3100implementing one or more embodiments. The system3100includes one or more processors3111(e.g., ASIC, CPU, etc.), and may further include an electronic display device3112(for displaying graphics, text, and other data), a main memory3113(e.g., random access memory (RAM), cache devices, etc.), storage device3114(e.g., hard disk drive), removable storage device3115(e.g., removable storage drive, removable memory, a magnetic tape drive, optical disk drive, computer-readable medium having stored therein computer software and/or data), user interface device3116(e.g., keyboard, touch screen, keypad, pointing device), and a communication interface3117(e.g., modem, wireless transceiver (such as Wi-Fi, Cellular), a network interface (such as an Ethernet card), a communications port, or a PCMCIA slot and card).

The communication interface3117allows software and data to be transferred between the computer system and external devices through the Internet3150, mobile electronic device3151, a server3152, a network3153, etc. The system3100further includes a communications infrastructure3118(e.g., a communications bus, cross bar, or network) to which the aforementioned devices3111through3117are connected.

The information transferred via communications interface3117may be in the form of signals such as electronic, electromagnetic, optical, or other signals capable of being received by communications interface3117, via a communication link that carries signals and may be implemented using wire or cable, fiber optics, a phone line, a cellular phone link, a radio frequency (RF) link, and/or other communication channels.

In one implementation of one or more embodiments in a mobile wireless device (e.g., a mobile phone, tablet, wearable device, etc.), the system3100further includes an image capture device3120, such as a camera128(FIG.2), and an audio capture device3119, such as a microphone122(FIG.2). The system3100may further include application processing or processors as MMS3121, SMS3122, email3123, social network interface (SNI)3124, audio/video (AV) player3125, web browser3126, image capture3127, etc.

In one embodiment, the system3100includes content curation and display management processing3130that may implement processing similar as described regarding visual app129(FIG.2) and processing content curation, display device management as described above. In one embodiment, the content curation and display management processing3130along with an operating system3129may be implemented as executable code residing in a memory of the system3100. In another embodiment, the content curation and display management processing3130may be provided in hardware, firmware, etc.

One or more embodiments provide intuitive and easy curation of display devices (e.g., display devices140,FIG.2) in a visual manner, and may function by tracking the images in the display device itself. The one or more embodiments do not require additional visual markers such as AR patterns, bar codes and QR codes. Advanced interactions, such as drag and drop are employed to assign content to display devices. Applying special effects to specific display devices is provided via advanced spatial interactions. Spreading an image across multiple display devices are provided in a spatially coherent manner. Selecting important regions/features of interest in an image to be displayed to specific display devices is provided. The one or more embodiments enable spatially intelligent editing of images based on the size and aspect ratio of the display device that the image is going to be assigned to.

The one or more embodiments provide virtual representations of a cluster of display devices as virtual walls, which may be saved, retrieved and curated even when geographically separated from the physical displays with the same user experience. Dynamic creation of groups of displays and their characteristics/behavior is provided. Several types of group interactions via unique UI mechanisms are provided. Suggestions for selection of filters based on neighboring image content and ambience of the display devices such as physical wall color and lighting is provided. Suggestions for the most suitable photos/content either from a local collection or curated services based on ambient information is provided.

The one or more embodiments provide suggestions of various templates and patterns of arrangement by detecting the number of display devices to be mounted and their various shapes and sizes, and perform calculations and provide precise measurements to achieve a selected pattern. Real-time position and alignment feedback is provided and corrections and guidelines are also provided in order to achieve a selected pattern. A far-away perspective of a cluster of displays is provided, even when standing close by to them, which assists with decisions about positioning and alignment. The environment is analyzed to mimic its features such as colors, texture and patterns within digital mat regions, resulting in blending of the digital frames with the environment and pleasing visual effects.

As is known to those skilled in the art, the aforementioned example architectures described above, according to said architectures, can be implemented in many ways, such as program instructions for execution by a processor, as software modules, microcode, as computer program product on computer readable media, as analog/logic circuits, as application specific integrated circuits, as firmware, as consumer electronic devices, AV devices, wireless/wired transmitters, wireless/wired receivers, networks, multi-media devices, etc. Further, embodiments of said Architecture can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment containing both hardware and software elements.

One or more embodiments have been described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to one or more embodiments. Each block of such illustrations/diagrams, or combinations thereof, can be implemented by computer program instructions. The computer program instructions when provided to a processor produce a machine, such that the instructions, which execute via the processor create means for implementing the functions/operations specified in the flowchart and/or block diagram. Each block in the flowchart/block diagrams may represent a hardware and/or software module or logic, implementing one or more embodiments. In alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures, concurrently, etc.

The terms “computer program medium,” “computer usable medium,” “computer readable medium”, and “computer program product,” are used to generally refer to media such as main memory, secondary memory, removable storage drive, a hard disk installed in hard disk drive. These computer program products are means for providing software to the computer system. The computer readable medium allows the computer system to read data, instructions, messages or message packets, and other computer readable information from the computer readable medium. The computer readable medium, for example, may include non-volatile memory, such as a floppy disk, ROM, flash memory, disk drive memory, a CD-ROM, and other permanent storage. It is useful, for example, for transporting information, such as data and computer instructions, between computer systems. Computer program instructions may be stored in a computer readable medium that can direct a computer, other programmable data processing apparatus, or other devices to function in a particular manner, such that the instructions stored in the computer readable medium produce an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.

Computer program instructions representing the block diagram and/or flowcharts herein may be loaded onto a computer, programmable data processing apparatus, or processing devices to cause a series of operations performed thereon to produce a computer implemented process. Computer programs (i.e., computer control logic) are stored in main memory and/or secondary memory. Computer programs may also be received via a communications interface. Such computer programs, when executed, enable the computer system to perform the features of the embodiments as discussed herein. In particular, the computer programs, when executed, enable the processor and/or multi-core processor to perform the features of the computer system. Such computer programs represent controllers of the computer system. A computer program product comprises a tangible storage medium readable by a computer system and storing instructions for execution by the computer system for performing a method of one or more embodiments.