Electronic device and operation method thereof

An electronic device and an operating method thereof are disclosed according to embodiments. An electronic device according to an embodiment includes: a camera; a display; a communication interface; a memory for storing one or more instructions; and a processor that executes the one or more instructions stored in the memory, the processor being configured to: generate an event instructing to display one or more markers on a screen of a display device and transmit the event to the display device; control the camera to capture an image of at least a portion of a wall around the display device using the one or more markers displayed on the screen of the display device according to the generated event; generate a background image to be displayed on the screen of the display device based on the obtained image; and transmit the generated background image to the display device such that the generated background image is displayed on the screen of the display device.

TECHNICAL FIELD

Various embodiments relate to an electronic device and an operation method thereof, and more particularly, to an electronic device and a method of operating the electronic device to display an image of a wall on a display device.

BACKGROUND ART

An image display device is a device having a function of displaying an image that a user can view. The user can view a broadcast through the image display device. The image display device displays a broadcast, which is selected by the user from among broadcast signals transmitted from a broadcast station, on a display. Currently, broadcasting has been changed worldwide from analog broadcasting to digital broadcasting.

Digital broadcasting refers to broadcasting digital images and audio signals. Compared with analog broadcasting, digital broadcasting is strong against external noise and has small data loss, is advantageous for error correction, has high resolution, and provides clear images. Also, unlike analog broadcasting, digital broadcasting can provide a bidirectional service.

Provided is a smart television providing various pieces of content in addition to a broadcasting function. The smart television is capable of analyzing and providing what a user desires without a user's operation, rather than being manually operated according to a user's selection.

In addition, recently, a product such as a wall-mounted television installed on a wall or a framed television for giving an aesthetic sense such as a picture frame has been provided to enhance the interior decoration of an environment in which the television is installed.

DESCRIPTION OF EMBODIMENTS

Technical Problem

Various embodiments are intended to provide a method and a device for displaying a wall image on a television screen to enhance an aesthetic sense of an environment in which a television is installed.

Solution to Problem

To solve this technical problem, an electronic device and an operation method thereof control a display device to display one or more markers on a screen of the display device in order to naturally display an image of a wall around the display device on the screen of the display device, capture the image of the wall around the display device using the displayed markers, and generate a background image to be displayed on the screen of the display device using the captured image.

Advantageous Effects of Disclosure

When a display device is installed according to embodiments, an image on a wall may be captured and processed more effectively to generate a background image to be displayed on the display device, thereby increasing an aesthetic sense of an environment in which the display device is installed.

BEST MODE

An electronic device according to an embodiment includes: a camera; a display; a communication interface; a memory for storing one or more instructions; and a processor that executes the one or more instructions stored in the memory, the processor configured to: generate an event instructing to display one or more markers on a screen of a display device and transmit the event to the display device; control the camera to capture an image of at least a portion of a wall around the display device using the one or more markers displayed on the screen of the display device according to the generated event; generate a background image to be displayed on the screen of the display device based on the obtained image; and transmit the generated background image to the display device such that the generated background image is displayed on the screen of the display device.

According to an embodiment, the processor is configured to execute the instructions to: obtain size information about the screen of the display device; and determine a position on which the one or more markers will be displayed on the screen of the display device based on the obtained size information and generate the event including information about the determined marker display position.

According to an embodiment, the processor is configured to execute the instructions to: control the camera to capture an image of at least a portion of the wall around the display device using the one or more markers displayed on the screen of the display device and a preview image displayed on the display.

According to an embodiment, the processor is configured to execute the instructions to: control the display to display the preview image including one or more guides in consideration of a position of at least a portion of a wall to be captured; and control the camera to capture an image of at least a portion of the wall as the one or more guides included in the preview image are positioned on the one or more markers displayed on the screen of the display device.

According to an embodiment, at least a partial area of a wall on which the display device is installed may include at least one of a left area, a right area, an upper area, and a lower area of a wall adjacent to the display device.

According to an embodiment, the processor is configured to execute the instructions to: generate a background image by combining one or more copies of the obtained image and one or more copies of a symmetric image of the obtained image.

According to an embodiment, the processor is configured to execute the instructions to: generate a background image by combining the obtained image with a symmetric image of the obtained image when the size of the obtained image is half the size of the background image.

According to an embodiment, the processor is configured to execute the instructions to: when the obtained image is an image obtained from a left area or a right area of a wall adjacent to the display device, the processor, obtain the symmetric image by symmetrically arranging the obtained image in the left or right direction of the obtained image; and when the obtained image is an image obtained from an upper area or a lower area of the wall adjacent to the display device, obtain the symmetric image by symmetrically arranging the obtained image in the upper or lower direction of the obtained image.

According to an embodiment, the processor is configured to execute the instructions to: transmit a marker display event and a background image to the display device through a server.

According to an embodiment, the processor is configured to execute the instructions to: control the camera to obtain the background image which is displayed on a screen of the display device and a peripheral area of the display device; adjust an attribute of the background image displayed on the screen using an attribute of the peripheral area; and transmit the background image with the adjusted attribute to the display device.

A method of operating an electronic device according to an embodiment includes: generating an event instructing to display one or more markers on a screen of a display device and transmitting the event to the display device; capturing an image of at least a portion of a wall around the display device using the one or more markers displayed on the screen of the display device according to the generated event; generating a background image to be displayed on the screen of the display device based on the obtained image; and transmitting the generated background image to the display device such that the generated background image is displayed on the screen of the display device.

According to an embodiment, disclosed is a non-transitory computer-readable recording medium having recorded thereon a program for executing a method of operating a display device in a computer.

MODE OF DISCLOSURE

Terms used in the present specification will be briefly described, and the present inventive concept will be described in detail.

General and widely used terms have been employed herein, in consideration of functions provided in the disclosure, and may vary according to an intention of one of ordinary skill in the art, a precedent, or emergence of new technologies. Additionally, in some cases, an applicant may arbitrarily select specific terms, in which case, the applicant will provide the meaning of the terms in the description of the embodiments. Accordingly, it will be understood that the terms used herein should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

Throughout the specification, when a portion “includes” an element, another element may be further included, rather than excluding the existence of the other element, unless otherwise described. In addition, terms such as “ . . . unit”, “ . . . module”, or the like refer to units that perform at least one function or operation, and the units may be implemented as hardware or software or as a combination of hardware and software.

The disclosed example embodiments will be described with reference to the accompanying drawings in such a manner that the example embodiments may be easily carried out by those of ordinary skill in the art. In this regard, the present disclosure may have different forms and should not be construed as being limited to the descriptions set forth herein. In addition, descriptions of well-known functions and constructions will be omitted for clarity and conciseness, and similar reference numerals are assigned to similar elements throughout the specification.

In embodiments of the specification, the term “user” refers to a person who controls functions or operations of an image display device using a control device, and may include a viewer, a manager, or an installer.

FIG. 1is a reference diagram for describing a concept of displaying a wall image on a display device, according to embodiments.

Referring toFIG. 1, a display device200is installed on a wall10. The display device200according to an embodiment may display an image11of a portion of the wall10in a state other than a normal operation mode of the display device200that displays content according to a user input. In a state in which a user does not use the display device200by operating the same, the display device200may display the image11of a portion of the wall10to increase an aesthetic sense of an environment in which the display device200is installed. In addition, by making a bezel201of the display device200slimmer or by implementing the bezel201in a color not recognized by human eyes, it is possible to provide a user with an experience as if the display device200is not on the wall image11while the display device200is not operating. A TV providing such an effect may be referred to as a glass TV.

As such, when the display device200displays the wall image11, the more the real wall10and the wall image11are expressed so that they may be naturally connected, a user may experience the display device200as if it were a wall. After the display device200is installed on the wall10, it is difficult to identify a portion of the wall10covered by the display device200. Therefore, it is important to express the wall image11displayed by the display device200in a state where the display device200is installed on the wall10so as to be connected to a peripheral area of the display device200as naturally as possible.

FIG. 2is a reference diagram for describing a method of displaying a wall image on a display device, according to an embodiment.

Referring toFIG. 2, the display device200installed on the wall10may display at least one marker21. The display device200may display a marker according to a request from a mobile device100. The display device200may display a marker by receiving information about one of a display position of the marker, that is, one of the right, left, top, and bottom of a screen from the mobile device100.FIG. 2corresponds to a case in which a request to display a marker on the right side of the screen is received from the mobile device100.

The mobile device100may provide the display device200with the information about the display position of the marker to the display device200. In the case ofFIG. 2, the mobile device100provides a request to display the marker21on the right area on the display device200. At this time, in order to capture a right wall image of the display device200, the mobile device may display a guide23on the left area of the screen of the mobile device100.

The mobile device100may receive a selection from a user as to which area of peripheral walls of the display device200is to be captured. For example, when a user selects a right wall area as an area to be captured among the peripheral walls of the display device200, the mobile device100may transmit information to the display device200to display the marker21on the right side of a screen of the display device so as to capture the right wall area of the display device. In addition, the guide23for matching with a marker displayed on the display device200may be displayed on the left side of the screen of the mobile device100.

The mobile device100may display one or more guides on a display of the mobile device100. The mobile device100may capture an image of a portion of the wall10using at least one guide23and at least one marker21displayed on the display device200. For example, when a user moves the mobile device100so that the guide23of the mobile device100is positioned at the marker21displayed on the display device200, the mobile device100may capture a wall image22. When the guide23is positioned at the marker21, because the mobile device100receives an image of a right peripheral area22of the display device200in a right portion24of an area captured by the mobile device100, the mobile device100may capture the right peripheral area22of the wall10. Because the area22is a right peripheral area of the display device200, the mobile device100may generate a wall image to be displayed on the display device200, that is, a background image, by performing appropriate processing using an image of the captured area22. In addition, because the area22is an area immediately adjacent to the display device200, when a background image to be displayed on the display device200is generated using the image of the area22, an image that appears to be more naturally connected to the peripheral area of the display device200may be generated.

InFIG. 2, the right area22of the display device200to be captured is an example, and the captured area may be any position of the left area, the upper area, and the lower area of the display device200.

FIG. 3is a schematic view for describing an example of a system for displaying a wall image on a display device, according to embodiments.

Referring toFIG. 3, according to an embodiment, the system may include the mobile device100, the display device200, and a server300. The mobile device100, the display device200, and the server300may communicate with each other through a communication network30. The communication network30may include a broadband network according to various communication standards such as Zigbee, 3rd Generation (3G), 3rd Generation Partnership Project (3GPP), and Long Term Evolution (LTE).

The mobile device100may receive information about the display device200from the display device200or the server300through the server300, specifically screen size information, and may generate a marker display event based on the information.

The marker display event is transmitted to the display device200through the server300, and when the marker display event is received, the display device200may display one or more markers on a screen. The markers may be displayed at various positions on the screen of the display device200.

The mobile device100may capture an image of a wall around the display device200using a marker displayed on the display device200. The mobile device100may capture an image of an area around the display device200according to a position of the marker displayed on the screen of the display device200. For example, when a marker is displayed on the left half screen of the display device200, the mobile device100may capture an image on an area adjacent to the left side of the display device200.

The mobile device100may generate a background image to be displayed on the display device200by using the captured original image and a symmetric image that mirrors the original image, and may transmit the generated background image to the display device200through the server300.

When the background image is received through the server300, the display device200may display the background image on the screen.

FIG. 4is a schematic view for describing another example of a system for displaying a wall image on a display device, according to embodiments.

Referring toFIG. 4, according to an embodiment, the system may include the mobile device100and the display device200. The mobile device100and the display device200may communicate with each other through a short-range communication method. The short-range communication method may use short-range communication protocols such as Bluetooth, Bluetooth low energy (BLE), Wi-Fi, Wi-Fi Direct, Near Field Communication (NFC), infrared communication, and laser beam communication.

Because the mobile device100does not go through the server, the mobile device100may receive screen information about the display device200directly from the display device200and may directly transmit a marker display event generated thereby to the display device200.

The display device200displays at least one marker on the screen according to the marker display event, and the mobile device100may capture a peripheral wall image of the display device200using a marker displayed on the screen of the display device200, may generate a background image using the captured image, and may transmit the generated background image to the display device200.

The display device200may receive the background image and display the same on the screen.

FIG. 5is a block diagram of the mobile device100, the display device200, and the server300according to an embodiment.

The display device200may display an image by processing content data received through various sources. The display device200may receive an RF signal from a broadcasting station using an RF antenna. The display device200may receive content provided by a content providing server through a communication network based on the Internet protocol. The display device200may receive content from external devices connected to the display device200through various ports or HDMI. The external devices may include, for example, set-top boxes, game consoles, BD players, PCs, USBs, and the like. In addition, the display device200may receive content from a mobile device that is connected and communicates with the display device200, and may also obtain content data from a memory that is self-contained or locally connected.

The display device200may include a display210, a communication interface220, a sensor230, and a controller240.

The display210displays content received through various sources on a screen under the control of the controller240. The various sources may include, for example, the communication interface220, a memory250, and an input/output unit. The display210may be implemented as PDP, LCD, OLED, a flexible display, or the like, and may also be implemented as a three-dimensional (3D) display. Also, the display130may be configured as a touch screen and used as an input device in addition to an output device.

The communication interface220may connect the display device200to an external device, for example, the mobile device100or the server300, under the control of the controller240. The communication interface220may include one or a combination of wireless LAN, Bluetooth, and wired Ethernet according to the performance and structure of the display device. The communication interface220may further include short-range communication other than Bluetooth, for example, near field communication (NFC) and Bluetooth low energy (BLE).

The sensor230may include a microphone that detects a user's voice, a user's video, or user's interaction, and receives a user's uttered voice, a camera unit capable of receiving an image corresponding to a user's motion including a gesture in a camera recognition range, and an optical receiver receiving an optical signal (including a control signal) received from an external remote control device according to a user input. The sensor230detects a command or a control signal from the remote control device through the optical receiver, and transmits the detected command or control signal to the controller240. The remote control device may be implemented as various types of devices for controlling the display device200such as a remote controller or a mobile phone.

The memory250may store various data, programs, or applications for driving and controlling the display device200under the control of the controller240. The memory250may store an operating system for controlling the display device200and the controller, an application initially provided from a manufacturer or downloaded from outside, a graphical user interface (GUI) related to the application, objects (e.g., images, texts, icons, buttons, etc.) for providing the GUI, user information, documents, databases, or related data.

The memory250includes a memory card (e.g., a micro SD card, a USB memory; not shown) mounted in ROM, RAM, or the display device200. In addition, the memory200may include a nonvolatile memory, a volatile memory, a hard disk drive (HDD), or a solid state drive (SSD).

In particular, according to an embodiment, the memory250may include one or more instructions for transmitting information about the display device200at a request of the mobile device100or the server300. The information about the display device200may include size information of the display device200, specifically screen size information.

According to an embodiment, the memory250may include one or more instructions for receiving a marker display event from the mobile device100or the server300and displaying one or more markers on the display according to the marker display event.

According to an embodiment, the memory250may include one or more instructions for receiving a background image from the mobile device100or the server300and displaying the background image on the display. The background image may include an image generated using a peripheral area of a wall on which the display device200is installed.

The controller240controls all operations of the display device200and a signal flow between internal components of the display device200and performs data processing. The controller240may execute an operating system (OS) and various applications stored in the memory250when the controller240receives a user input or satisfies conditions already set and stored.

A processor used in the controller240may include a graphics processing unit (GPU) (not shown) for graphics processing corresponding to a video. The processor may be implemented as a System On Chip (SaC) integrating a core (not shown) and the GPU (not shown). The processor may include a single core, a dual core, a triple core, a quad core, and multiples thereof.

In addition, the processor may include a plurality of processors. For example, the processor may be implemented as a main processor (not shown) and a sub processor (not shown) operating in a sleep mode.

The controller240according to an embodiment may transmit information about the display device200according to the request of the mobile device100or the server300by executing one or more instructions stored in the memory250.

The controller240according to an embodiment may receive a marker display event from the mobile device100or the server300by executing one or more instructions stored in the memory250, and may display one or more markers on the display according to the marker display event.

The controller240according to an embodiment may receive a background image from the mobile device100or the server300by executing one or more instructions stored in the memory250, and may display the background image on the display.

The display device200may further include an input/output unit that receives video (e.g., a moving picture, etc.), audio (e.g., voice, music, etc.), and additional information (e.g., an EPG, etc.) from the outside of the display device200under control of the controller240. The input/output unit may include one or a combination of a High-Definition Multimedia Interface (HDMI) port, a component jack, a PC port, and a USB port. The display device200according to an embodiment may be a TV, but this is only an embodiment and may be implemented as an electronic device including a display. For example, the display device200may be implemented as various electronic devices such as a desktop computer or a digital broadcasting terminal. The display device200may be a flat display device, a curved display device having a curved screen, or a flexible display with adjustable curvature.

Furthermore, the display device200may be electrically connected to a separate external device (e.g., a set-top box, not shown) having a tuner. For example, the display device200may be implemented as an analog TV, a digital TV, a 3D TV, a smart TV, an LED TV, an OLED TV, a plasma TV, a monitor, or the like, but is not limited thereto.

Meanwhile, the block diagram of the illustrated display device200is a block diagram for an embodiment. Each component of the block diagram may be integrated, added, or omitted depending on the specification of the actual display device200. Each component of the block diagram may be integrated, added, or omitted, depending on specifications of the image display device200that is actually implemented. For example, if necessary, two or more components may be combined into one component, or one component may be divided into two or more components. In addition, the functions performed by the respective blocks are for purposes of describing the embodiments, that the specific operation and the device shall not limit the scope of the present disclosure.

The server300is a computing device in which a server program providing a service is executed at the request of a user (client).

The server300may include a communication interface310and a controller320.

The communication interface310may connect the server300to an external device, for example, the mobile device100or the display device200, under the control of the controller320. The communication interface220may include one or a combination of wireless LAN, Bluetooth, and wired Ethernet according to the performance and structure of the display device.

The memory330may store various data, programs, or applications for driving and controlling the server300under the control of the controller320. The memory330may store an operating system for controlling the server300and the controller, an application initially provided from a manufacturer or downloaded from outside, a graphical user interface (GUI) related to the application, objects (e.g., images, texts, icons, buttons, etc.) for providing the GUI, user information, documents, databases, or related data.

The memory330includes a memory card (e.g., a micro SD card or a USB memory; not shown) mounted in ROM, RAM, or the server300. In addition, the memory330may include a nonvolatile memory, a volatile memory, a hard disk drive (HDD), or a solid state drive (SSD).

In particular, according to an embodiment, the memory330may include one or more instructions for transmitting an application for controlling one or more devices in the home to the mobile device100at the request of the mobile device100. A device control application may be used to control the display device200, and particularly, may include one or more instructions for setting a background image of the display device200.

According to an embodiment, the memory330may include one or more instructions for transmitting information about the display device200at the request of the mobile device100. The information about the display device200may include size information of the display device200, specifically screen size information.

According to an embodiment, the memory330may include one or more instructions for receiving a marker display event from the mobile device100and transmitting the marker display event to the display device200.

According to an embodiment, the memory330may include one or more instructions for receiving a background image from the mobile device100and transmitting the background image to the display device200. The background image may include an image generated using a peripheral area of a wall on which the display device200is installed.

The controller410controls all operations of the server300and a signal flow between internal components of the server300and performs data processing. The controller320may execute an operating system (OS) and various applications stored in the memory250when the controller320receives a user input or satisfies conditions already set and stored.

A processor used in the controller320may include a graphics processing unit (GPU) (not shown) for graphics processing corresponding to a video. The processor may be implemented as a System On Chip (SoC) integrating a core (not shown) and the GPU (not shown). The processor may include a single core, a dual core, a triple core, a quad core, and multiples thereof.

In addition, the processor may include a plurality of processors. For example, the processor may be implemented as a main processor (not shown) and a sub processor (not shown) operating in a sleep mode.

The memory330according to an embodiment may transmit an application for controlling one or more devices in the home to the mobile device100by executing one or more instructions stored in the memory330. The device control application may be used to control the display device200, and particularly, may include one or more instructions for setting the background image of the display device200.

The controller320according to an embodiment may transmit information about the display device200according to the request of the mobile device100by executing one or more instructions stored in the memory330. The information about the display device200may include size information of the display device200, specifically screen size information.

The controller320according to an embodiment may receive a marker display event from the mobile device100by executing one or more instructions stored in the memory330, and may transmit the marker display event to the display device200.

The controller320according to an embodiment may receive a background image from the mobile device100by executing one or more instructions stored in the memory330, and may transmit the background image to the display device200. The background image may include an image generated using a peripheral area of a wall on which the display device200is installed.

The mobile device100may capture an image of a wall on which the display device200is installed, may generate a background image to be displayed on a screen of the display device200based on the captured image, and then may transmit the background image to the display device200through the server300.

The mobile device100may include an imaging sensor circuit110, a communication interface120, a display130, a memory140, and a controller150.

The imaging sensor circuit110may receive an image (e.g., continuous frames) corresponding to user's motion including a gesture in a recognition range of an imaging sensor. The imaging sensor circuit110may be implemented as a camera. The imaging sensor circuit110according to an embodiment may capture an image representing a wall to be displayed by the display device200.

The communication interface120may connect the mobile device100to an external device, for example, the display device200or the server300, under the control of the controller150. The communication interface120may include one or a combination of wireless LAN, Bluetooth, and wired Ethernet according to the performance and structure of the mobile device. The communication interface120may specifically include a mobile communication unit and a sub-communication unit.

The mobile communication unit performs broadband network communication according to various communication standards such as Zigbee, 3rd Generation (3G), 3rd Generation Partnership Project (3GPP), and LTE (Long Term Evolution). The mobile communication unit may be used for communication in the server300.

The sub-communication unit communicates with peripheral devices using short-range communication protocols such as Bluetooth, Bluetooth low energy (BLE), Wi-Fi, Near Field Communication (NFC), infrared communication, and laser beam communication. The sub-communication unit may be used for communication with the display device200.

The display130displays content received through various sources on a screen under the control of the controller150. The various sources may include, for example, the communication interface120, a memory140, and an input/output unit. Also, the display130may be configured as a touch screen and used as an input device in addition to an output device.

The memory140may store various data, programs, or applications for driving and controlling the mobile device100under the control of the controller150. The memory140may store an operating system for controlling the mobile device100and the controller, an application initially provided from a manufacturer or downloaded from outside, a graphical user interface (GUI) related to the application, objects (e.g., images, texts, icons, buttons, etc.) for providing the GUI, user information, documents, databases, or related data.

The memory140includes a memory card (e.g., an SD card, a USB memory; not shown) mounted in ROM, RAM, or the mobile device100.

In particular, according to an embodiment, the memory140may include one or more instructions for transmitting a download request of an application for controlling a home device to the server300, and receiving and storing the device control application accordingly. The device control application may include one or more instructions for controlling the display device200, particularly one or more instructions used to set the background image of the display device200.

According to an embodiment, the memory140may include one or more instructions for transmitting information about the display device200to the mobile device100or the server300. The information about the display device200may include size information of the display device200, specifically screen size information.

According to an embodiment, the memory140may include one or more instructions for generating a marker display event using information about the display device200and transmitting the marker display event to the mobile device100or the server300. The marker display event may include position information of one or more markers to be displayed on the screen of the display device according to a screen size of the display device200.

According to an embodiment, the memory140, according to the marker display event, may include one or more instructions for capturing a partial image of a wall on which the display device200is installed using one or more markers displayed on the screen of the display device200, processing the captured image to generate a background image to be displayed on the screen of the display device200, and transmitting the background image to the display device200or the server300.

The controller150controls all operations of the mobile device100and a signal flow between internal components of the mobile device100and performs data processing. The controller150may execute an operating system (OS) and various applications stored in the memory140when the controller150receives a user input or satisfies conditions already set and stored.

A processor used in the controller150may include a graphics processing unit (GPU) (not shown) for graphics processing corresponding to a video. The processor may be implemented as a System On Chip (SoC) integrating a core (not shown) and the GPU (not shown). The processor may include a single core, a dual core, a triple core, a quad core, and multiples thereof.

In addition, the processor may include a plurality of processors. For example, the processor may be implemented as a main processor (not shown) and a sub processor (not shown) operating in a sleep mode.

The controller150according to an embodiment may transmit a request to download an application for controlling a device in the home to the server300by executing one or more instructions stored in the memory140, and may receive and store a device control application accordingly. The device control application may include one or more instructions for controlling the display device200, particularly one or more instructions used to set the background image of the display device200.

The controller150according to an embodiment may transmit information about the display device200to the mobile device100or the server300by executing one or more instructions stored in the memory140. The information about the display device200may include size information of the display device200, specifically screen size information.

The controller150according to an embodiment, by executing one or more instructions stored in the memory140, may generate a marker display event using information about the display device200, and may transmit the marker display event to the mobile device100or the server300. The marker display event may include position information of one or more markers to be displayed on the screen of the display device according to the screen size of the display device200.

The controller150according to an embodiment, by executing one or more instructions stored in the memory140, may capture a partial image of a wall on which the display device200is installed using one or more markers displayed on the screen of the display device200according to a marker display event, may process the captured image to generate a background image to be displayed on the screen of the display device200, and may transmit the background image to the display device200or the server300.

Meanwhile, the block diagram of the illustrated mobile device100is a block diagram for an embodiment. Each component of the block diagram may be integrated, added, or omitted depending on the specification of the mobile device100that is actually implemented. For example, combining two or more elements are in a single component, as needed, or may be one component configuration is subdivided into two or more components. In addition, the functions performed by the respective blocks are for purposes of describing the embodiments, that the specific operation and the device shall not limit the scope of the present disclosure.

The image mobile device100according to an embodiment may be realized as various electronic devices such as a cellular phone, a tablet PC, a digital camera, a camcorder, a laptop computer, an e-book terminal, a terminal for digital broadcasting, a personal digital assistant (PDA), a portable multimedia player (PMP), a navigation, an MP3 player, and an wearable device.

FIG. 6illustrates a method of operating a mobile device according to an embodiment.

Referring toFIG. 6, in operation S610, the mobile device100generates an event instructing that one or more markers are displayed on a screen of the display device200and transmits the event to the display device200.

According to an embodiment, the mobile device100may determine a position of the one or more markers to be displayed on the screen of the display device200by referring to screen size information of the display device200.

According to an embodiment, the mobile device100may receive the screen size information of the display device200from a user, the display device200, or the server300.

According to an embodiment, the mobile device100may generate a marker display event including the determined position of the marker and transmit the marker display event to the display device200.

According to an embodiment, the mobile device100may directly transmit the marker display event to the display device200or may transmit the marker display event to the display device200through the server300.

In operation S620, the mobile device100captures an image of at least a portion of a wall around the display device200using one or more markers displayed on the screen of the display device according to the generated marker display event.

According to one embodiment, the mobile device100may capture an image of at least a portion of a wall around the display device200using one or more markers displayed on the screen of the display device and one or more guides displayed on the display of the mobile device.

In operation S630, the mobile device100generates a background image to be displayed on the screen of the display device200based on the captured image.

According to an embodiment, the mobile device100may generate a background image using a captured original image and a symmetric image of the captured original image.

In operation S640, the mobile device100transmits the generated background image to the display device such that the generated background image may be displayed on the screen of the display device.

According to an embodiment, the mobile device100may transmit the generated background image directly to the display device200or through the server300.

FIG. 7is a flowchart illustrating an operation of a mobile device, a server, and a display device for displaying a background image on the display device using an image of a wall on which the display device is installed, according to an embodiment.

Referring toFIG. 7, in operation S705, the mobile device100downloads and installs a device control application, and in operation S710, the server300transmits the device control application to the mobile device100.

When the mobile device100requests to download the device control application to the server300, the server may transmit the device control application to the mobile device100, and the mobile device100may receive and install the device control application. The device control application may include applications for controlling various devices in the home, for example, a refrigerator, a display device, a washing machine, an audio device, and the like.

In operation S715, the mobile device100may execute the installed device control application.

In operation S720, the mobile device100may receive a user input for selecting a background screen setting menu.

FIG. 8illustrates a user interface provided by the mobile device100by executing a device control application, according to an embodiment.

Referring toFIG. 8, a first user interface810of the device control application executed by the mobile device100may receive a user input for selecting a device to be controlled. The first user interface810may display a display device811, an audio device812, a washing machine813, and a refrigerator814as the device to be controlled, and a user may select one of the displayed devices.

When the display device811is selected by the user, the mobile device100may output a second user interface820. The second user interface820may display menu items related to control of the display device, and one of the items may include a background screen setting item821. The background screen setting item821is a menu item that enables setting to display the same image as the image on a wall on which the display device is installed while the display device is not operated by a user operation.

When the background setting item821is selected by the user, the mobile device100may output a third user interface830. The third user interface830may include an input window831through which a display device model name may be input. The mobile device100may receive the display device model name through the input window831.

Again, in operation S725, the mobile device100may request and obtain screen information of the display device. The screen information of the display device includes the size of a display screen of the display device200. The screen information of the display device may be used to determine positions of one or more markers to be displayed on the screen of the display device. The screen information of the display device may be included in product information of the display device, and may be obtained using the display device model name.

The mobile device100may obtain the screen information of the display device using the display device model name received through the third user interface830.

In operation S730, according to an embodiment, the mobile device100may receive screen information of the display device corresponding to the display device model name from the server300by transmitting the display device model name to the server300.

In operation S735, according to another embodiment, the server300may transmit the display device model name received from the mobile device100to the display device200, and the display device200may transmit screen information of the display device corresponding to the display device model name to the mobile device100through the server300.

According to another example, because the mobile device100stores screen information of the display device corresponding to the display device model name, the mobile device100may obtain screen information of the display device without requesting the server300.

In operation S740, the mobile device100may select an area to be captured among images of a wall on which the display device200is installed. After obtaining the screen information of the display device, the mobile device100may be changed to a photo shooting mode.

When the mobile device100is positioned so that a user may obtain an image of the wall on which the display device200is installed, a camera of the mobile device100may sense the image of the wall on which the display device200is installed, including the display device200, and a display of the mobile device100may display the image of the wall including the display device200as a preview image. Also, the mobile device100may display an outline of an area to be captured on the preview image. In addition, the mobile device100may receive an input for selecting one area from a user by displaying outlines of a plurality of areas to be captured. A method of selecting an area to be captured by the mobile device100will be described with reference toFIGS. 9 to 11.

FIG. 9illustrates an example of a preview image displayed on a display of the mobile device100, according to an embodiment.

Referring toFIG. 9, the mobile device100displays an image of a wall on which the display device200is installed as a preview image900on the display. Also, the mobile device100may display a guideline for selecting an area to be captured on the displayed preview image900. InFIG. 9, a guideline910indicating a right area A, a guideline920indicating an upper area B, and a guideline930indicating a left area C are displayed on the preview image900. A user may select an image area of a wall to be captured by selecting one of outlines910,920, and930.

The mobile device100may variously determine an area of a wall to be captured.

According to an embodiment, the mobile device100may internally determine an area of a wall to be captured as a default.

According to an embodiment, the mobile device100may display areas of a wall to be captured as shown inFIG. 9and allow a user to select one of the areas.

According to an embodiment, the mobile device100may further include a <Complete> item950in addition to a guideline for selecting an area to be captured in the preview image900for selecting an area to be captured.

For example, when a user selects A, one of A, B, and C as an area to be captured on the preview image900, and presses a <Finish> item950, the mobile device100may transmit information to the display device200to display a marker on the right side of the display device screen such that the right wall area C of the display device may be captured, and may also display a guide for matching with a marker displayed on the display device200on the left side of a screen of the mobile device100.

FIG. 10is a view illustrating an area to be captured among peripheral walls of a display device, according to an embodiment.

1000A ofFIG. 10illustrates an example of capturing wall images of the right, upper, and left areas of peripheral walls of the display device200when the display device200is a stand type, according to an embodiment. When a screen size of the display device200is (horizontal X, vertical Y), the size of a left area or a right area to be captured may be determined as (horizontal X/2, vertical Y), and the size of an upper area to be captured may be determined as (horizontal X, vertical Y/2). Each of area A, area B, and area C corresponds to half the screen size of the display device200. Therefore, by doubling the size of the captured area, a background image to be displayed on a screen of the display device200may be generated.

1000B ofFIG. 10illustrates an example of capturing wall images of the left, right, upper, and lower areas of peripheral walls of the display device200when the display device200is wall-mounted, according to an embodiment.

FIG. 11is a reference diagram for describing a size of a peripheral wall area of a display device to be captured, according to an embodiment.

1100A ofFIG. 11illustrates an example of capturing wall images of the left, right, upper, and lower areas of peripheral walls of the display device200, according to an embodiment. When a screen size of the display device200is (horizontal X, vertical Y), the size of a left area or a right area to be captured may be determined as (horizontal X/3, vertical Y), and the size of an upper area or lower area to be captured may be determined as (horizontal X, vertical Y/3). Each of area A, area B, area C, and area D corresponds to ⅓ of the screen size of the display device200. Therefore, by making the size of a captured area three times larger, a background image to be displayed on a screen of the display device200may be generated.

1100B ofFIG. 11illustrates an example of capturing wall images of the left, right, upper, and lower areas of peripheral walls of the display device200, according to an embodiment. When a screen size of the display device200is (horizontal X, vertical Y), the size of a left area or a right area to be captured may be determined as (horizontal X/4, vertical Y), and the size of an upper area or lower area to be captured may be determined as (horizontal X, vertical Y/4). Each of area A, area B, area C, and area D corresponds to ¼ of the screen size of the display device200. Therefore, by making the size of a captured area four times larger, a background image to be displayed on a screen of the display device200may be generated.

In operation S745, the mobile device100generates a marker display event and transmits the same to the server300. In operation S750, the server300may transmit the marker display event received by the mobile device100to the display device200. In operation S755, the display device200may display one or more markers on a display screen according to the marker display event received from the server300.

Specific operations of operations S745, S750, and S755will be described with reference toFIGS. 12 to 13.

FIG. 12is a reference diagram for describing a method of a mobile device to transmit a marker display event to a display device, according to an embodiment.

Referring toFIG. 12, the mobile device100may generate a marker display event1200using the screen information of the display device obtained in operation S725and the captured area selected in operation S740. The marker display event is an event that causes the display device200to display one or more markers on a display screen. The marker display event may include position information of one or more markers to be displayed on the display screen along with a command to cause the display device200to display one or more markers on the display screen. The mobile device100may transmit the generated marker display event1200directly to the display device200or through the server300. The display device200that has received the marker display event1200may display one or more markers1201,1202,1203, and1204on the display screen according to marker display position information included in the marker display event. The marker display position information is changed according to the captured area selected in operation S740. As illustrated inFIG. 12A, in order to capture a right wall image i.e. an image of right wall adjacent to the display device200, one or more markers may be displayed on the right half of a display screen of the display device200.

In the example shown inFIG. 12, markers are displayed in a square shape, but the markers may be displayed in various shapes such as a circle or a triangle.

FIG. 13is a view for explaining a method of determining a marker display position according to a selected captured area, according to an embodiment.

Referring toFIG. 13, when area A is selected as an area to be captured in the mobile device100, an area of a corresponding wall is area A′. The mobile device100may determine a position to display one or more markers on the right half of a screen of the display device200to capture an image of the area A′. Accordingly, when the area A is selected, the mobile device100may determine marker display positions as (m1, m2, m3, and m4). Also, when area B is selected, the mobile device100may determine marker display positions as (m1, m6, m9, and m10) to display one or more markers on the upper half of the screen of the display device. Also, when area C is selected, the mobile device100may determine marker display positions as (m5, m6, m7, and m8) to display one or more markers on the left half of the screen of the display device.

In the example illustrated inFIG. 13, because an area to be captured is determined as half of the entire display device screen, one or more markers are displayed at positions corresponding to half of the display device screen. However, when the area to be captured is ⅓ or ¼ of the entire display device screen, positions of the markers are also changed accordingly.

In addition, in the example shown inFIG. 13, although it has been exemplified to display four markers on the screen of the mobile device100, this is only an example, and the markers may be one or more.

Returning to S760again, the mobile device100may capture a wall image using one or more markers displayed on the display device200.

According to an embodiment, as shown inFIG. 12, when markers are displayed on the display device200, a user sees the marker and moves the mobile device100such that the markers are positioned on the left half of the screen of the mobile device100, and when a right area1210of the display device200is positioned in the right half on a preview image of the mobile device100, the mobile device100may capture an image according to a user's image capture command.

According to another embodiment, when a user sees markers on the display device200and moves the mobile device100such that the markers are positioned on the left half of the screen of the mobile device100, an imaging sensor of the mobile device100recognizes one or more markers displayed on the display device200, and when recognition is completed, the mobile device100may automatically capture a wall image. The mobile device100may recognize a marker by comparing a feature point of a previously stored marker image with a feature point extracted from a marker displayed on the display device200using a marker recognition image processing technology through feature point matching.

According to another embodiment, the mobile device100may display one or more guides on a preview image displayed on a display screen of the mobile device100to more accurately capture images. When a user moves the mobile device100such that a guide on the preview image is positioned on one or more markers displayed on the display screen of the display device200, the imaging sensor of the mobile device100may capture a wall image when one or more markers are recognized. This will be described with reference toFIGS. 14 to 16.

FIG. 14is a reference diagram for explaining an example of displaying one or more guides on a mobile device, according to an embodiment.

Referring toFIG. 14, the mobile device100may display one or more guides1401,1402,1403, and1404on a preview image to help recognition of the markers1201,1202,1203, and1204displayed on a screen of the display device200. To enable the mobile device100to capture the right wall image1210adjacent to the display device200, the display device200may display a marker on the right half screen of the display device200and display a guide on the left half on the preview image of the mobile device100. When a user positions the guides1401,1402,1403,1404on the preview image to the markers1201,1202,1203, and1204displayed on the screen of the display device200, respectively, the right wall image1210of the display device200may enter a right half area1220of the preview image displayed on the screen of the mobile device100. Therefore, in this state, the right wall image1210may be obtained by capturing an image as the imaging sensor of the mobile device100recognizes the markers1201,1202,1203, and1204.

In addition, in order to provide help to a user, the mobile device may display a message1400<Place a guide displayed on a screen on a marker displayed on the display device> on the preview image.

In addition, in the example illustrated inFIG. 14, the number of markers displayed on the screen of the mobile device100is four, but this is only an example, and the number of markers may be one or more.

FIG. 15is a reference diagram for explaining an example of displaying one or more guides on a mobile device, according to an embodiment.

Referring toFIG. 15, the mobile device100may display one or more guides1511,1512,1513, and1514on a preview image to help recognition of markers1501,1502,1503, and1504displayed on a screen of the display device200. To enable the mobile device100to capture an upper wall image1510i.e. an image of an upper wall adjacent to the display device200, the display device200may display a marker on the right half screen of the display device200and may display the guides1511to1514on the left half on the preview image of the mobile device100. When a user positions the guides1511,1512,1513, and1514on the preview image to the markers1501,1502,1503, and1504displayed on the screen of the display device200, respectively, an upper wall image1500adjacent to the display device200may be captured within the upper half region1510of the preview image displayed on the screen of the mobile device100. Therefore, in this state, the imaging sensor of the mobile device100may capture an image to obtain the upper wall1500.

In operation S765again, the mobile device100generates a background image using the captured wall image.

The mobile device100may generate a background image by processing the captured wall image.

According to an embodiment, in order to create a natural background image that can be more harmonious with a wall, the mobile device100, considering a bezel thickness d of the display device200as shown inFIG. 16, may obtain a wall image from an area spaced apart by d from the end of the bezel of the display device200, rather than a wall image directly adjacent to the display device200.

Meanwhile, the mobile device100may obtain images of a plurality of peripheral areas, and then provide a user interface to allow a user to select a desired area.

FIG. 17is an example of a user interface capable of selecting one of a plurality of area images, according to an embodiment.

Referring toFIG. 17, the mobile device100may provide a user interface1600that obtains all images for peripheral areas A, B, C, and D, displays obtained area A image1630, area B image1610, area C image1640, and area D image1620on the display screen, and selects one of these area images. By providing the user interface1600, a user may select a desired image from all possible wall images. In particular, as shown inFIG. 17, when there is a pattern in a left wall area C and a right wall area A of the display device200, it is important to select an image of the area C or the area A as a background image to be displayed on the display device200.

When a captured area image to be used for the background image is determined, the mobile device100may generate a background image using the captured area image.

FIG. 18is a reference diagram for explaining a method of generating a background image using a ½ captured image of a screen, according to an embodiment.

Referring toFIG. 18, when an original image1810captured by the mobile device100from a left wall area of the display device200is ½ of a display screen of a display device, the mobile device100may generate a background image by doubling the size of the original image1810. In this case, referring toFIG. 18, when the mobile device100captures the image1810on a left wall area1800adjacent to the display device200and uses the captured original image1810to generate a background image to be displayed on the display device200, it may be important to generate a background image1840such that the left wall area1800and a background image1840are naturally connected. For example, it is important to show that an area1801adjacent to the display device200in the left wall area1800and an area1841adjacent to the left wall area1800in the background image1840are naturally connected. Therefore, for this purpose, the mobile device100may position an area1801in the area1841of the background image1840by placing a first image1820generated by symmetrically centering a right edge of the captured original image1810in the left half of the background image1800. By placing a second image1830symmetrically centered on a right edge of the first image1820again in the right half of the background image1840, an image of an area1821may be positioned in an area1831.

By generating the background image using a mirror image of the captured original image1810, it is possible to generate the background image1840in which the areas1801and1841are naturally connected to each other and the areas1821and1831are naturally connected to each other.

FIG. 19is a reference diagram for explaining a method of generating a background image using a ¼ captured image of a screen, according to an embodiment.

Referring toFIG. 19, when an original image1910captured by the mobile device100from a left wall area of the display device200is ¼ of a display screen of a display device, the mobile device100may generate a background image by making the size of the original image1910four times. At this time, the mobile device100may generate a background image1960by placing a first image generated by symmetrically centering a right edge of the captured original image1910in a first area1920of the background image, by placing a second image generated by symmetrically centering a right edge of the first image in a second area1930of the background image, by placing a third image generated by symmetrically centering a right edge of the second image in a third area1940of the background image, and by placing a fourth image generated by symmetrically centering a right edge of the third image in a fourth area1950of the background image.

Meanwhile, a user may check a background image generated by the mobile device100before applying the background image to the display device.

FIG. 20illustrates an example of a user interface capable of erasing a portion to be removed from a background image generated according to an embodiment.

Referring toFIG. 20, the mobile device100may display the generated background image on a display. For example, as illustrated inFIG. 2, when the mobile device100captures the area22to generate a background image, a portion of a flower in a vase may be displayed under an image of the captured area22, and thus some shapes2011and2012of the flower may be displayed on a generated background image2010. A user may remove the shapes2011and2012from the background image2010by touching or scrubbing the some shapes2011and2012of the flower using a finger or other tool. To remove some of the shapes displayed on the image using a finger or a tool, a known picture editing technique or the like may be used.

Returning to operation S770again, the mobile device100may transmit the generated background image to the display device200.

According to an embodiment, the mobile device100may directly transmit the generated background image to the display device200.

According to an embodiment, the mobile device100may transmit the generated background image to the display device200through the server300. In operation S775, the server300may transmit the background image to the display device200, and in operation S780, the display device200may receive the background image and display the background image on a display screen.

According to an embodiment, after the display device200receives and displays the background image, a user may want to change the color tone or brightness of the background image displayed on the display device200. For example, the color tone of the background image displayed on the display device200may be different from the color tone of an actual wall. Therefore, in this case, the mobile device100may further provide a user interface that allows a user to change the color tone or brightness of the background image.

FIG. 21illustrates an example of checking a background image displayed on the display device200and changing a color of the background image by a user through the mobile device100, according to an embodiment.

Referring toFIG. 21, the display device200displays a background image2110received from the mobile device100. In addition, the mobile device100may provide a user interface2120to change the color of the background image. When a user adjusts the color of the background image through the user interface2120, the mobile device100may transmit the color-adjusted background image back to the display device200.

According to an embodiment, when a wall on which the display device200is installed is a wall without any pattern, the mobile device100may generate a background image using a representative color of the wall on which the display device200is installed.

FIG. 22is a view for explaining an example of generating a background image using a representative color of a wall, according to an embodiment.

Referring toFIG. 22, when a wall on which the display device200is installed is recognized as a patternless wall having no pattern, the mobile device100may extract a representative color from an image obtained by capturing a peripheral wall area2200and generate a background image using the representative color and a screen size of the display device. That is, the mobile device100may generate a background image by setting an area corresponding to a background image size of the display device to a representative color.

According to an embodiment, when the wall on which the display device200is installed is a wall in which a certain pattern is repeated, the mobile device100may generate a background image using a representative pattern of the wall on which the display device200is installed.

FIG. 23is a view for explaining an example of generating a background image using a representative pattern of a wall, according to an embodiment.

Referring toFIG. 23, when a wall on which the display device200is installed is recognized as a wall in which a uniform pattern is repeated uniformly, the mobile device100may extract a representative pattern from an image obtained by capturing a peripheral wall area2300and generate a background image using the representative pattern and a screen size of the display device. That is, the mobile device100may generate a background image by repeatedly arranging a representative pattern in an area corresponding to a background image size of the display device.

A method of operating a mobile device, a display device, and a server according to an embodiment may be implemented as program commands which can be executed by various computer means, and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures or a combination thereof. Program instructions recorded on the medium may be particularly designed and structured for the present disclosure or available to those of skill in computer software. Examples of the computer-readable recording medium include magnetic media (e.g., hard discs, floppy discs, or magnetic tapes), optical media (e.g., CD-ROMs or DVDs), magneto-optical media (e.g., floptical discs), and hardware devices that are specially configured to store and carry out program commands (e.g., ROMs, RAMs, or flash memories). Program commands may include, for example, high-level language code that can be executed by a computer using an interpreter, as well as machine language code made by a compiler.