Layout Recognition for Dynamic Contents Retrieval in Digital Signage
Adiyan Mujibiya*

Abstract. We report our approach to bridge dynamic content transfer from publicly available digital signage tousers’ private smart devices, such as glass-like wearables and smartphones. We aim to address issues concerning
dynamic multimedia signage that has its contents divided into several sections. This type of signage has become
increasingly popular due to optimal content exposures. In contrast to prior research, our approach excludes
computer vision (CV) based object recognition, and instead took a novel approach to identify how contents arebeing laid-out in a digital signage. We incorporate CV techniques to recognize basic layout features such ascorners and line segments, which are obtained from camera frame of user’s device. Consequently, the visualfeatures are combined to generate content layout map, which will be compared to pre-learned layout map toaggregate corrected perspective. To grab a specific content, users are able to choose a section within the captured
layout using the device’s interface, which in turn creates a request to contents server to send respective content
info based on a timestamp and a unique section ID. We describe implementation details, report preliminary results,
and conclude with discussion of our experiences in implementation as well as highlighting our future plans.

1Introduction

In recent years digital signage has penetrated
significant numbers of public spaces and graduallyreplacing traditional printed or electric (bulbs/LED)
billboards. This trend is mainly driven by the decreasing
cost to deploy large LCD displays that are practicallysuitable to visualize dynamically changing and varied
information. The common practice to maximize contents
exposure in digital signage is to divide the screen canvas
into several sections respective to content’s type or genre.
These multi-section signage offers clean layout designand support simultaneous contents delivery.
Research aiming to close the gap between public
signage and user’s private mobile devices has been
conducted very actively in HCI field [1,2,4]. However, tothe best of our knowledge, there is no work thatspecifically addresses contents bridging for multi-section
digital signage. QR codes as well as other 2D/3D
barcodes are still widely used to provide users with entry
access point to detailed information. However, visual
codes and URLs are incompatible for multi-section
signage due to spatial and design constraints.
In this abstract, we report our approach to address
aforementioned issues by implementing signage layout
recognition based on CV techniques, which includes
corner and line segment detection [3] of images captured
Copyright is held by the author(s).
*Rakuten Institute of Technology, Rakuten, Inc.
" %!#Figure 1. We report our approach to realize contents transferfrom multi-section digital signage to users’ private device, suchas glass-like system and smartphones. We use computer vision
approach, which includes corner and line segment detection toidentify signage layout within user’s camera view.

on the user’s device. Corners and line segments are
combined with other 2D features to generate content
layout map, which is then matched with pre-learned
layout map to aggregate corrected perspective. To grab a
specific content, users are able to choose a section within
the captured layout using the device’s interface, which inturn creates a request to contents server to send
respective info based on a timestamp and a uniquesection ID. Actual usage scenario is depicted in Figure 1.
Using this approach, we mitigate requirement topre-learn contents for section-specific object recognition,
thus allowing dynamic changes of contents in eachsection, which in real-world practices may include
seasonal scheduling, real-time updates, changing
contents source, and so on. Moreover, our layout

WISS 2014

Figure 2. We describe our signage layout recognition system. First, we define contents layout to be trained using its corners and linesegments that serves as a groundtruth of our layout matching. Second, we fill the contents that are positioned according to the pre-defined
static layout. Here, the contents in each section can be changed dynamically. Lastly, we perform layout recognition using a camera on
user’s device, perform perspective correction, and let the user select a section which s/he wants to receive further information from.

recognition performs in real-time at up to 8 fps for VGA
resolution (640×360), deployed in a glass-like wearable
device with OMAP4430 2GB RAM dual-core SoC, on
Android 4.4.2 operating system.

2

Proposed Method

We highlight our proposed method in Figure 2. We
define a static layout and assign a unique ID for eachsection. We aggregate features such as corners andrelative positioning of corners, and line segments. We
use these features to form content layout map, which isbeing used to match features aggregated from camera
frames, and obtain corrected layout perspective.
Based on the static layout, we fill the contents using
standard web technologies. In our prototype, we
incorporate 11 sections in our layout, and assign contentssuch as café poster slideshow, food and beverage menu
slideshow, special offer slideshow, news update text feed
and video-cast, as well as weather cast.
We implement client-server protocol to realize layout
recognition and content distribution. Our layout
recognition software was implemented mostly in Java,
with some parts in C++ for computationally resource
consuming functions. We utilize glass-like wearable astarget device, considering that the device is equippedwith a see-through optical head-mount display that willemphasize user experience on public-to-private contents
retrieval. An exemplary figure of user’s view is depicted
in Figure 2 (right). The green outline visualizes
perspective corrected signage layout, and sections arehighlighted in color shades. To select a section, user canbrowse through sections using the glass device’s touch
interface. Finding appropriate interface design toaccommodate section selection for specific device

remains future work. After user selection has been
confirmed, the client software creates a request to thecontent distribution server to send back a URL that refers
to details or further information related to selected
section. The server side of our system also manages
contents scheduling, therefore pairing between
timestamps to layout section IDs and its contents arestraightforward.

3Discussion and Future WorkWe define content layout map based on line segments,
which are locally straight zones of the image where thegrey level is changing significantly. Therefore, ourapproach requires visible frame design (observable inFigure 2, depicted as black pixels in signage background).
This potentially become a visual design constraint that
limits signage designers when creating aestheticalelements in their contents. Exploring layout recognition
without aforementioned constraint remains future work.

Reference[1]

She, J., Crowcroft, J., Fu, H., and Li, F.. Convergence ofinteractive displays with Smart mobile devices for
effective advertising: A survey. ACM Trans. Multimedia
Comput. Commun. Appl.10, 2 (2014), Article 17.a.

[2]

Turner, J. Cross-device eye-based interaction. In UIST’13,37-40.

[3]

Von Gioi, R.G., Jakubowicz, J., Morel, J.M., Randall, G.
LSD: a Line Segment Detector. In Image Processing On
Line, 2 (2012), pp. 35–55.

[4]

Yamaguchi, T., Fukushima, H., Tatsuzawa, S., Nonaka,M., Takashima, K., and Kitamura, Y.. SWINGNAGE:
Gesture-based mobile interactions on distant public
displays. In ITS’13, 329-332.