System and method for automatic video scaling

A method of identifying and scaling a region of interest on a display device is presented. The region of interest is detected based on the rate of change between frames. A size of the display area of the display device and aspect ratio are determined. The detected region of interest is scaled to fit the display area in full screen mode based on the display area and the aspect ratio. The region of interest may be, for example, an active video region. In full screen mode, any static or slow-moving images are hidden.

BACKGROUND

With the growth of Internet streaming services and online media sites such as YouTube, Netflix, Hulu, and Amazon, among others, wireless streaming technologies are becoming ever more popular, allowing users to stream digital media from a network-attached device to a display device. Wireless streaming technologies such as MiraCast, WiDi, ChromeCast, AirPlay are supported by a wide range of televisions, sticks, dongle, and set-top boxes, and are able to stream media from network-attached devices to a display device without the hassle of establishing wired connections. The network-attached device may include mobile phones, tablets, or smart TV. Video source information from the network-attached device is cast to a display device, which may also have wireless capabilities.

In some cases, video content from streaming services include an active video region surrounded by a background image, which is either a static image or a series of slowly updating images. In many, if not most cases, the user is only interested in viewing the content of the active video region and is not interested in viewing the surrounding background image, yet the active video region is not scaled to full screen.

A method for identifying an active video region of streaming video and automatically scaling the active video region is desired.

SUMMARY

A method of identifying and scaling a region of interest on a display device is presented. The region of interest is detected based on the rate of change between frames. A size of the display area of the display device and aspect ratio are determined. The detected region of interest is scaled to fit the display area in full screen mode based on the display area and the aspect ratio.

DETAILED DESCRIPTION

The detailed description set forth below, in connection with the appended drawings, is intended as a description of exemplary embodiments of a system and method for detecting and scaling up a region of interest to display in full screen mode. The description sets forth the features of the inventive technique in connection with the illustrated embodiments. It is to be understood, however, that the same or equivalent functions and structures may be accomplished by different embodiments that are also intended to be encompassed within the spirit and scope of the disclosure. As denoted elsewhere herein, like element numbers are intended to indicate like elements or features.

Embodiments of the present inventive concept provide a system and method for automatic video scaling. The disclosure pertains to a technique for processing input data to identify a region of interest (ROI) from a video content in a display system. The disclosure provides a method for determining a video region to be scaled and a zoom factor from the detected ROI to output display. The disclosure provides a method for automatically adjusting video center and zoom factor according to size of the display area, such that region of interest is mapped to the whole display area in full screen mode. The region of interest may be a region that shows active video content. The disclosure includes a method for accurately detecting the region of interest, for example based on a rate of data change using accumulated difference data. The disclosure also includes a method for applying the region of interest to the full display size with aspect ratio adjustment. In some embodiments, a user interface for selecting the ROI and adjusting region of interest to full display area size is provided.

FIG. 1depicts the functional relationship among components of an example wireless display system.

A sender2, which transmits video and/or audio data to be displayed, has capabilities to be attached to a network, such as the Internet, via wireless or wired means. The sender2is further capable of being configured to handle wireless communication with a receiver4by implementing a wireless streaming technology. For example, the sender2may be configured to operate using ChromeCast (Google Cast), AirPlay, MiraCast, WiDi, or any other like technology. The sender2may include a server, a computer, a mobile device such as a smartphone or a tablet, or any other device with the above-described capabilities. The sender2is often not physically connected to the receiver4.

The sender2may include a sender app running on the sender device. The sender app may allow a user to select which display device content is to be displayed on, have media controls such as play/pause/record functionalities, and/or allow content discovery by the user.

The receiver4may receive digital media from the sender. The digital media may include video data (e.g. in a format such as MPEG2 or MPEG4, or the like) and/or audio data (e.g. in a format such as MP3, AAC, AC3, or the like) which is to be streamed. The receiver4may be configured to operate using a wireless streaming technology (as listed above for the sender2) corresponding to the wireless display technology used by the sender2. The receiver4may be a device such as a Chromecast dongle, Apple TV, a personal computer.

The receiver4may include a decoder6. The decoder6may include codecs included on a system on a chip (SoC) which is contained in the receiver. The codecs have the capability to decode video/audio compression formats of the video/audio data. After the codecs decode the compressed video/audio data so that the video/audio data becomes uncompressed, the receiver4may feed the uncompressed video/audio data to a display processor8.

The display processor8may be configured to perform video enhancement and other processing related to displaying the video. The video is then output to the display device12via a secure digital/analog connection10(e.g. a HDCP compliant connection).

FIG. 2includes a flowchart depicting an embodiment of region of interest detection process. The input from video decoder6allows the system to receive an uncompressed video frame. The system accumulates differences between a plurality of frames.

As shown inFIG. 2, the system receives at least two uncompressed video frames that are sequentially generated from the receiver (20). Then, the system accumulates differences between the at least two uncompressed video frames (22). This difference accumulation process (22) is detailed further below in reference toFIG. 3.

Subsequent to accumulating differences (22), the system determines if there is enough difference data to proceed with identifying the region of interest (24). In one embodiment, the system makes this determination by comparing the value stored in the accumulated difference buffer with a difference threshold. The difference threshold is a predefined, selected value which corresponds to a high enough difference data value to identify a region of interest. By comparing the value of the accumulated difference buffer with the difference threshold, the system helps ensure that the system has enough information to identify the region of interest. If the system determines there is not enough difference data to proceed, then the system loops back to receive another uncompressed video frame.

According to another embodiment, the system does not calculate the accumulated difference, but instead proceeds once a threshold number of frames have been displayed. However, this method with frame counts may be affected by the amount of activity in the content of video stream. For example, if the video being displayed has a very static image(s), the threshold number of frames would be adjusted up.

If the system determines there is enough difference data to proceed, then the system proceeds down a left path of the flow chart and a right path of the flow chart. Parts of the left and right branches of the flowchart may be executed sequentially or simultaneously. On the left path, the system performs edge detection on the accumulated difference data to obtain an edge binary image output (26). Then, Hough transform is applied to the edge binary image output (28) to obtain lines from the edge binary image output. In one embodiment, the system detects only lines (e.g., straight lines) extending at an angle of around 0 and 90 degrees with respect to a first direction, where the first direction may be parallel to a longer side of the video. However, the present inventive concept is not limited thereto. For example, the system may assume that the region of interest may be displayed with a straight-line boundary that extends at an arbitrary angle with respect to the first direction.

On the right path of the flow chart ofFIG. 2, the system performs corner detection on the accumulated difference data to obtain potential corner locations (32). Then, the system matches up the potential corner locations with the detected lines (30) from the edge binary image output (34), and applies a rectangle rule to eliminate potential corner locations that are not matched with a line (36). The rectangle rule dictates that the corner candidates selected must form a rectangle. Thus, each corner candidate must have two perpendicular lines, provided by the Hough transform, and two counter corners, the lines. The two counter corners are corners located at an endpoint of each respective line, where each endpoint is not located at the vertex of the two perpendicular lines. After determining the corner candidates, the system identifies a rectangle outlining a region of interest candidate and determines the region of interest (38). The system can then scale the region of interest using the display area spec that was previously received and displays the scaled region of interest to the full display area (40).

FIG. 3depicts a flowchart that shows details of the difference accumulation process22between frames. The difference accumulation process22, in one aspect, measures the rate of data change over a number of frames. Stages50through56are part of the process22shown inFIG. 2. Once the system receives the uncompressed video frame, the system extracts data from each of the R, G, and B channels of the current frame at time t (50). The data of each current channel R(t), G(t), and B(t) is respectively subtracted from the data of each last channel R(t-1), G(t-1), and B(t-1) channel (herein also referred to as data of a first prior frame t-1for respective R, G, and B channels), and each difference (Diff) is added to a difference buffer Diff (52). At this point, the parameter “Accumulated Difference” may be a sum of the differences between a number of previous channels, for example R(t-2), R(t-3), and R(t-4) for the R channel (which are herein also referred to as data of a second, third, and fourth prior frames for the R channel). This “Accumulated Difference” is updated to include the latest Diff (54). If a static image is being displayed, the Accumulated Difference may stay roughly zero for a large number of frames. However, but if a video is being displayed, the Accumulated Difference may grow rapidly. Once the difference buffer Diff is added to the Accumulated Difference, each of the current values R(t), G(t), and B(t) are redefined as values of the previous channels R(t-1), G(t-1), and B(t-1) to extract updated current values from the channels (56).

FIG. 4depicts a region of interest scaling process according to one embodiment. After identifying a region of interest candidate (60, or38inFIG. 2), a region of interest candidate border is shown as a rectangle (62) and the user is asked whether the detected region is correct to scale to full screen (64). If the region of interest candidate is not the correct region to scale, the system will look for an additional region of interest candidate (68). If the system finds an additional candidate, the system will display the additional candidate to the user for selection. On the other hand, if the region of interest candidate is the correct region to scale to full screen, the system will confirm the aspect ratio of the display device (70) and adjust the aspect ratio of the video if necessary (72). The system may determine the aspect ratio based on the device type, which may be detected or provided by the user. The region of interest is then scaled to be displayed full screen on the display device (74).

Although not depicted, if the region of interest is tilted (such that no edge extends parallel to the edge of the display device), the system will perform a rotation to align the region of interest either following or preceding scaling of the region of interest to full screen.

Referring toFIGS. 5A, 5B, and 5Cvisually depict an embodiment of region of interest detection.FIG. 5A, for example, provides a visual depiction of a region of interest94within a full display area92, which may be a display area of a display device. In this particular figure, the region94having a high Accumulated Difference is shown to be brighter than the other parts of the display area92, therefore indicating that the bright region94is the region of interest. (i.e., the white areas correspond to a region of interest and the black areas correspond to static images).FIG. 5Amay be thought of as depicting the process that happens at stage22ofFIG. 2.

FIG. 5Bshows example edge binary image output from edge detection, such as edge binary image output produced from stage26ofFIG. 2, for example. As the process moves forward to determine the region of interest, the edges/lines around the rectangular area will be highlighted as a candidate for scaling up (stage62ofFIG. 4).FIG. 5Cdepicts the case where a user confirms that the highlighted rectangular area is indeed the region of interest, and the region is scaled up to fit the entire display area92. In the scaled-up version, the data that is shown outside the active video area inFIG. 5AandFIG. 5Bremain hidden. As mentioned above, in one embodiment, the scaling entails identifying a center of the region of interest and aligning it with a center of the display area, then applying a zoom factor that is determined mathematically based on the dimensions of the region of interest and the aspect ratio of the display device.

FIG. 6depicts a functional block diagram of an embodiment of a display processor that may be used to implement the region of interest detection and automatic scaling processes described above. As shown, video frames are received by a Difference Accumulator80having a Buffer82. An Edge Detector82, a Corner Detector84, a Line Detector86, and a Region Detector88work together to determine a region of interest, as depicted inFIG. 2. A “detector”96, as used herein, includes at least one of the Edge Detector82, Corner Detector84, Line Detector86, and Region Detector88, as shown, for example, inFIG. 6. Furthermore, an Adjuster90may determine the aspect ratio and may adjust the aspect ratio if necessary. These detectors and the adjuster may be implemented as software or non-transitory computer-readable instructions that are stored in a medium. In various embodiments, operating system software of a system that includes the display processor may provide an operating environment for softwares executing in the computer system, and may coordinate activities of the components of the computer system.

Various embodiments of the system that includes the display processor may be implemented with or involve one or more computer systems. The computer system is not intended to suggest any limitation as to the scope of use or functionality of described embodiments. The computer system includes at least one processor and memory, one or more input devices, one or more output devices, and one or more communication connections. The memory may be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory, etc.), or combination thereof. In one embodiment, the memory may store software for implementing various embodiments of the disclosed concept.

To provide for interaction between a user and the display processor, embodiments can be implemented using a computer having a display device, e.g., a CRT (cathode ray tube), LCD (liquid crystal display), projection screen, OLED display, 3D display, etc. for displaying information to the participants. A touchscreen, a keyboard and a pointing device, e.g., a mouse or a trackball, by which a user can provide input to the computer are also provided. Other kinds of devices can be used to provide for interaction with participants as well; for example, feedback provided to the player can be any form of sensory feedback, e.g. visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, brain waves, other physiological input, eye movements, gestures, body movements, or tactile input. For example, any of the above methods may be used to make a “selection” of the region of interest by confirming the highlighted rectangular area.

While this specification contains many specifics, these should not be construed as limitations on the scope of the disclosure as a whole or of what can be claimed. Rather, the examples provided should be viewed as descriptions of features specific to particular embodiments. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features can be described above as being executed in certain combinations in certain order, one or more features from a disclosed combination may in some cases be omitted from the combination, and the claimed combination can be directed to a subcombination or variation of a subcombination.

Various embodiments of the present invention may be described in the general context of computer-readable media. Computer-readable media are any available media that may be accessed within a computer system. By way of example, and not limitation, within the computer system, computer-readable media include memory, storage, communication media, and combinations thereof.

It should be understood that the invention can be practiced with modification and alteration within the spirit and scope of the disclosure and the appended claims. The description is not intended to be exhaustive or to limit the inventive concept to the precise form disclosed.