# Designing a Support Tool for Creative Advertising by Mining Collaboratively Tagged Ad Video Content: The Architecture of PromONTotion

Katia Kermanidis, Manolis Maragoudakis, Spyros Vosinakis, Nikos Exadaktylos

► To cite this version:

Katia Kermanidis, Manolis Maragoudakis, Spyros Vosinakis, Nikos Exadaktylos. Designing a Support Tool for Creative Advertising by Mining Collaboratively Tagged Ad Video Content: The Architecture of PromONTotion. 9th Artificial Intelligence Applications and Innovations (AIAI), Sep 2013, Paphos, Greece. pp.10-19, 10.1007/978-3-642-41142-7_2. hal-01459623

HAL Id: hal-01459623

https://inria.hal.science/hal-01459623

Submitted on 7 Feb 2017

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Designing a Support Tool for Creative Advertising by Mining Collaboratively Tagged Ad Video Content: The Architecture of PromONTotion

Katia Kermanidis¹, Manolis Maragoudakis², Spyros Vosinakis³, Nikos Exadaktylos⁴

¹Department of Informatics, Ionian University, 49100 Corfu, Greece

²Department of Information and Communication Systems Engineering, University of the Aegean, 83200 Karlovasi, Samos, Greece

³Department of Product and Systems Design Engineering, University of the Aegean, 84100 Ermoupoli, Syros, Greece

⁴Department of Marketing, Technological Educational Institute of Thessaloniki, 57400 Sindos, Greece

kerman@ionio.gr, mmarag@aegean.gr, spyrosv@aegean.gr, nexadakt@mkt.teithe.gr

**Abstract.** Creative advertising constitutes one of the highest-budget enterprises today. The present work describes the architecture of PromONTotion, an innovative tool for supporting ad designers in setting up a new campaign. The content of existing ad videos is collaboratively tagged, while playing a multi-player web-based game, and the provided annotations populate the support tool-thesaurus, a hierarchical ontological structure. Annotators-players will also provide information regarding the impact the ads had on them. Data mining and machine learning techniques are then employed for detecting interdependencies and correlations between ontology concepts and attributes, and for discovering underlying knowledge regarding the product type, the ad content and its impact on the players. The support tool will make ad videos, terms, statistical information and mined knowledge available to the ad designer, a generic knowledge thesaurus that combines previous ad content with users' sentiment to help the creative process of new ad design.

**Keywords:** creative advertising, creativity support tool, serious games, collaborative annotation, video annotation, advertisement ontology

## 1 Introduction

Creative advertising describes the process of capturing a novel idea/concept for an ad campaign and designing its implementation. It is governed nowadays by significant budget allocations and large investments. Several studies have been published regarding the impact of advertising (Amos, Holmes and Strutton, 2008; Aitken, Gray and Lawson, 2008), as well as creativity in advertising (Hill and Johnson, 2004).

A number of creativity support tools have been proposed, to help ad designers come up with novel ideas for setting up a new campaign. Such tools, that could en-
hance the development of creative ideas, are highly beneficial for the advertising in-
dustry. They usually focus on forcing upon the advertiser a certain restricted way of
thinking, using creativity templates (Goldenberg et al., 1999). The methodology is
based on the hypothesis that total freedom is not the most efficient way for enhancing
the creative process, but constraining it with the use of a limited number of idea-
forming patterns. Expert decision making systems, like ADCAD (Burke et al., 1990),
have been proposed for triggering creative ideas. ADCAD relies on rules and facts
that are in reality quite hard to provide as input to the decision making tool. Janusian
wording schemata (the use of opposite words in taglines) have been used extensively
(Blasko and Mokwa, 1986) in advertising. IdeaFisher (Chen, 1999) is based on a da-
tabase of predefined concepts and associations between them. GENI (MacGrimmon
and Wagner, 1994) guides the user to make connections and transformations between
the entities of a brainstorming problem. Idea Expander (Wang et al., 2010) is a tool
for supporting group brainstorming by showing pictures to a conversing group, the
selection of which is triggered by the conversation context. Opas (2008) presents a
detailed overview of several advertising support tools.

Most of the aforementioned creativity support tools make use of static non-
expandable databases, term-relation dictionaries, hand-crafted associations and trans-
formations. Static, passive, expert-dependent knowledge models can hurt creativity
(Opas, 2008).

The present work describes the architecture and design challenges of
PromONTotion, a creative advertising support tool that incorporates

- collaboratively accumulated ad video content annotations

- associations between these annotations derived though reasoning within the onto-
logical structure they form

- knowledge concerning ad type, ad content, ad impact and the relations between them; the knowledge is extracted with mining techniques

- statistical information regarding the impact an ad video has on consumers.

Unlike previous approaches to creativity support tool design, PromONTotion relies on
no predefined or hand-crafted elements and associations, apart from an empty onto-
logical backbone structure, that will include ad content concepts, consumer impact
data slots, and taxonomic relations between them. The populating of the backbone, as
well as the remaining knowledge available by the tool, are data-driven and automati-
cally derived, making PromONTotion generic, dynamic, scalable, expandable, robust
and therefore minimally restricting in the creative process and imposing minimal
limitations to ideation or brainstorming.

The proposed tool faces a number of significant research challenges. First,
crowdsourcing will be employed for the collection of collaborative ad content tags.
Following the Games with a Purpose approach (von Ahn, 2006), a multiplayer brows-
er-based action game, ‘House of Ads’, is implemented for this particular purpose
focusing on challenging and entertaining gameplay in order to keep the players’ en-
gagement at a high level for a long time. The ontology is an innovation by itself; its
content (concepts, features, relations), coverage and representation are very interest-
ing research issues. Mapping the output of the game, i.e. the annotations, to the onto-
logical structure is a very intriguing task, as well as the end-user interface that needs
to be friendly and make full use of the resource capabilities. The interdisciplinary
nature of the proposed idea is challenging, as the areas of video game design, ontolo-
gy engineering, human-computer interaction and advertising are linked together to
produce an advertising support tool.

In the remainder of this paper, section 2 describes in detail the architecture and the
major design challenges of PromONTotion, i.e. the ontological backbone, the serious
videogame for content annotation, the data mining phase and the final creativity sup-
port tool. The paper concludes in the final section.

## 2 The Architecture of PromONTotion

The architecture design of the proposed tool relies on several distinct underlying
phases, components and techniques. Figure 1 shows the various elements and tasks of
PromONTotion.

<figure><img src="image_2.png" /><figcaption>Fig. 1. The Architecture of PromONTotion</figcaption></figure>

### 2.1 The Ontological Backbone

Marketing and ontology experts have sketched the ontological structure that will in-
clude concepts, categories and taxonomic (is-a, part-of etc.) relations between them,
that are relevant to an advertising campaign. The parameters that determine the mes-
sage to the consumer (e.g. tag lines), the ad filming technicalities, the ad content, as
well as its artistic features play a significant role in designing the semantic ontology.
Also, subjective information regarding the impact of the ad to the consumers is in-
cluded in the ontology. Table 1 shows a significant part of the ontological backbone.
<table><caption>**Table 1.** The ontological backbone</caption><thead><tr><th>Concept</th><th>Level 1<br>Sub-concepts</th><th>Level 2<br>Sub-concepts</th><th>Concept Terms/Values</th></tr></thead><tbody><tr><td>Filming<br>Director<br>Product Type</td><td>Product</td><td></td><td>studio/outdoors<br>director name<br>houseware/electronic/food/<br>luxury/store/other</td></tr><tr><td>Main Character</td><td>Service<br>Human</td><td>gender</td><td>telecom/banking/.../other<br>male/female</td></tr><tr><td>Key participating objects<br>Location</td><td>Animate<br>Fictitious</td><td>age<br>occupation/hobby/activity<br>recognisability<br>type of animal</td><td>age value<br>employee/businessman/<br>farmer/ housework /.../other<br>celebrity/everyday person<br>pet/wild/...<br>cartoon/comics/movie hero<br>tool/furniture/vehicle/electronic/technical/other<br>home/office/work/other<br>urban/rural/other</td></tr><tr><td>Genre</td><td></td><td></td><td>non-fiction/tale/cartoon/<br>animsted</td></tr><tr><td>Art</td><td>soundtrack<br>photography</td><td></td><td>rock/classical/ethnic/.../other<br>picturesque/landscape/<br>airphoto/other<br>colour palette</td></tr><tr><td>Message<br>Communication<br>Strategy</td><td>Linguistic schemata<br>Adoption of known<br>elements</td><td>taglines/word<br>games/paraphrasing<br>/Janusian wording<br>book lines</td><td>tagline text / existence of<br>paraphrasing/word games<br>known line text</td></tr><tr><td>Ad Impact</td><td>Indirect critique on<br>competition<br>Convincing power<br>Consumer sentiment</td><td>movie lines<br>song lines</td><td>known line text<br>known line text<br>yes/no<br>a lot/some/none<br>positive/neutral/negative</td></tr></tbody></table>

As shown in the table, the concepts are organized in a hierarchical taxonomy structure and are represented through a set of features/sub-concepts, thereby forming the desired ontology. For the development of the ontology, as well as the ability to per-
form reasoning on its content, an ontology editor will be employed, like Protégé<fnref n="1" /> or OntoEdit<fnref n="2" />. It is important for the ontology to be scalable, so it can constantly be enriched and updated.

## 2.2 The ‘House of Ads’ Game

In order to support the generic, minimal human-expertise-demanding and data-driven nature of the proposed support tool, the ontology backbone is populated through crowdsourcing techniques. It is evident that the success of PromONTotion relies heavily on the plethora of provided annotations; therefore the annotation tool needs to be attractive, engaging, fun and addictive. To this end, ‘House of Ads’, a browser-based game, is designed and implemented especially for the task at hand.

Several toolkits exist for annotating text (Wang et al., 2010; Chamberlain et al., 2008), images, like Catmaid, Flickr, Riya, ImageLabeler and Imagenotion (Walter and Nagypal, 2007), or video, like VideoAnnex and YouTube Collaborative Annotations have been proposed. Von Ahn (2006) recognized that the high level of game popularity may be taken advantage of, and channeled towards other, more “serious”, applications, instead of only pure entertainment. Siorpaes and Hepp (2008) were the first to propose a game for ontology populating. The nature of textual data has not allowed for the design of genuinely entertaining gaming annotation software. The annotation of ad videos, however, inspires the design of software that can keep the player’s interest and engagement level active for a very long time. Unlike the Imagenotion ontology, the thesaurus aimed at by PromONTotion is comprised of a backbone of a more elaborate set of concepts and relations, as well as statistical information regarding the terms inserted by players to populate the backbone, requiring a more elaborate game platform. The design of engaging game scenarios with usable and attractive interfaces has been recognized as one of the key challenges in the design of Games with a Purpose for content annotation (Siorpaes and Hepp, 2008). In the design of ‘House of Ads’ a first step towards this goal is attempted by adding the fun elements of interaction and competition (Prensky, 2001) in the game and by including typical action-game challenges rather than simply adopting a quiz-like gameplay.

‘House of Ads’ is an arcade-style, top-down shoot-em-up, and puzzle-like game, accessible to anyone. It supports one to four players and includes two gameplay modes: the combat mode and the quiz mode (Figures 2 and 3 show some indicative screenshots of the two gameplay modes). In the former, ontology concepts and subconcepts are mapped as rooms and each ontology term as a collectable game object within the room representing the respective concept. The atmosphere is retro-like, resembling the arcade-like games of the 80s and 90s. In a TV screen the ad video is reproduced (selected from the over 300,000 available ad videos on youtube), and the players, simulated as characters navigating within the house, aim at collecting as many objects that characterize the content of the ad as quickly as possible and exit the

<footnote n="1">¹ http://protege.stanford.edu</footnote>

<footnote n="2">² www.ontoknowledge.org/tools/ontoedit.shtml</footnote>
house. Players are free to collect another object from the same category if they believe that an object collected by others incorrectly describes the content with respect to this concept. Therefore, contradictory answers and possible cheaters can be easily spotted. The video reproduction may be controlled by the player (paused, forwarded etc.) through a slider, and every player can block other players from reaching their goal by using a set of available weapons (fence, bombs etc.). The house platform supports different levels of difficulty; as the game evolves the house becomes more complicated, with more rooms and more detailed elements, demanding more fine-grained annotations and more complex attack/defense strategies. At the end of each stage, the player will be asked to comment on the convincing ability and the impact of the ad, providing his personal opinion. There is a large number of available questionnaires for the evaluation of advertising campaigns<fnref n="3" /> <fnref n="4" />, based on which this type of information may be provided.

<figure><img src="image_3.png" /><figcaption>Fig. 2. The ‘House of Ads’ combat mode</figcaption></figure>

Players earn money at the end of each stage, which can be used to buy new weapons or to improve their abilities. For each object, that can be safely assumed by the system that it is correctly characterizing the ad, the money earned is free to be spent. For objects whose correctness cannot be immediately decided, the money is blocked and will be freed once a positive decision will be made in the future. If a player collects incorrect items, his/her credibility drops and, if it falls below a certain threshold, the game ends and the player has to start over. Annotation correctness is established

<footnote n="3">³ www.surveyshare.com/templates/televisionadvertisementvaluation.html</footnote>

<footnote n="4">⁴ www-sea.questionpro.com/akira/showSurveyLibrary.do?surveyID=119&mode=1</footnote>
by taking into account the total number of answers to a question (the greater this number, the safer the drawn conclusions), the majority vote, the popularity of the other votes, the player credibility. A wide popularity (acceptance) of the game will ensure correctness as well as completeness (the complete ontology populating). In-complete game sessions (sessions that have not provided annotations to all ontology slots) are not as important as attracting as many players as possible.

The quiz mode is dedicated to resolving annotation questions that could not be re-solved in the combat mode, e.g. contradictory answers. The questions and their possible answers are posed to all the players simultaneously, and the player who first selects the correct answer receives the money that was blocked so far.

The game will be available online and accessible by anyone. Social media and popular networks will be exploited for its dissemination and diffusion. The game will be evaluated according to several aspects. Its usability and engagement will be tested through interviews and questionnaires handed out to a group of players, in combination with the talk aloud protocol, used extensively for the evaluation of the usability of interfaces. Another evaluation aspect is whether the game does indeed manage to populate the proposed ontology. Further evaluation metrics to determine the success of the approach may include: mean times played and number of successful annotations per player, percentage of incorrect answers, etc.

<figure><img src="image_4.png" /><figcaption>Fig. 3. The ‘House of Ads’ quiz mode</figcaption></figure>

## 2.3 Data Mining

All player annotations are used to populate the ontology backbone and are stored in a database for further processing, that includes statistical as well as data mining techniques. The goal of the data processing phase is the detection of co occurrence and correlation information between categories, terms and relations (e.g. in how many ads for cleaning products there is a housewife in the leading role etc.), and its statistical significance, as well as higher-level association information governing them.

Data mining and machine learning techniques will also be employed. The concepts and attributes described earlier will enable the transformation of the annotation set of
a given advertisement into a learning feature-value vector. The vectors will, in turn, enable feature selection (through wrapping, filtering or combinations of the two), that will reveal the significance of the selected concepts on advertisement design (and indirectly evaluate the ontology design). Learning will reveal correlations between ad content choices, ad products and attempt prediction of consumer impact. The extracted, mined, knowledge will reveal very interesting and previously unknown information regarding the parameters that directly or indirectly affect ad design and play a role on the consumers' sentiment and how the latter may be influenced.

Due to the anticipated data sparseness and providing that a plethora of attributes will be collected, emphasis will be given to dimensionality reduction algorithms. Singular Value Decomposition (e.g. M. Lee et al., 2010) as well as Support Vector Machines (Vapnik, 1995) will be utilized in order to deal with the aforementioned issues. Visualization of instances and attributes on a new feature space, according to the most significant vectors will reveal clusters of similar ontology mappings together with advertised products. Support Vectors will be used to weight each attribute according to a certain criterion, such as correctness of user responses and could be used to evaluate the most significant ontology nodes. Finally, another potential use of data mining techniques for the task at hand will be that of association rules discovery, based on the FP-growth algorithm. Association rules are easily interpretable by humans and can be straightforwardly evaluated by them.

## 2.4 The Creativity Support Tool

The ontological structure with all its content, objective (content annotations) and subjective (ad impact annotations from every player), along with the derived correlations and extracted knowledge mined in the previous phase, will be made usable to professionals in the advertising domain through a user-friendly interface. Advertisers will be able to see the content of old ads for related products, and thereby come up with new ideas, gain insight regarding the impact of previous campaigns from the players' evaluation, look for screenshots of videos using intelligent search, based not only on keywords, but on concepts. More specifically, advertisers will be able

- to have access to a rich library of video ads

- to search the videos by content, based on a query of keywords (e.g. a specific type of product)

- to retrieve statistical data regarding the ads, i.e. see the terms/concepts/attributes his search keyword co-occurs with most frequently

- have access to the consumers' evaluation on the advertisements' impact.

The innovative generic nature of the tool will allow it to be flexible, scalable and adjustable to the end user's needs. Most importantly, unlike creativity templates, the generic nature does not impose any sort of 'mold' or template to the creative advertiser's way of thinking.

The support tool will be evaluated by a group of advertising experts, who will be handed a product and will be asked to create a hypothetical ad scenario for it. They will evaluate the support tool based on its usability, its completeness, its significance.
A combination of evaluation approaches will be employed in order to record the opinion and the impressions of the end users. Questionnaires will be handed out and interviews will be conducted to detect the problems and weaknesses of the support tool. Problems in the tool usability will be identified by evaluating its usage in real time with the think-aloud protocol. A group of ontology experts will evaluate the created ontology based on international ontology evaluation standards for its coverage, classification ability etc.

## 3 Conclusion

In this work the architecture and the major design issues of the creative advertising support tool PromONTotion have been presented. The tool will facilitate the brainstorming process of advertisers through a semantic thesaurus that includes video ad content annotations collected via crowdsourcing techniques, as well as knowledge relating to ad concepts, product type, annotation terms and the relations governing them. Annotations are objective, describing the content of the video, as well as subjective, denoting the annotator's personal sentiment towards the ad. The annotations will be collected through collaborative game playing. While playing, annotators will tag ad videos crawled from the web, describe the ad content and artistic features, and evaluate the ad impact on themselves. This information (the set of terms, concepts and subjective opinions), as well as statistical co occurrence data regarding concepts, advertised products, and subjective impact, will be structured into a hierarchical ontology. A user-friendly interface will allow ad designers to make full use of the ontology's capabilities, and advertising experts will evaluate the tool's coverage, usability and significance.

Several research questions still remain challenging for future work. The popularity of the game is a major concern, as it will define the number of collected annotations. Future prospects should be open to alterations and improvements on the game to increase its popularity. The use of other crowdsourcing techniques, like the Mechanical Turk, could form another future direction for gathering annotation data. Regarding the machine learning process, various mining techniques and learning schemata could be experimented with, so that the ones more suitable for the specific datasets can be established. The support tool usability is also of research significance, as the underlying knowledge needs to be transparent to the end user, and the tool needs to be as unobtrusive as possible in the creative process. Finally, regarding the impact of a specific ad to consumers, broader sentiment information could be extracted through web sentiment analysis of social media data, and complement the one provided by the annotators.

## Acknowledgements

This Project is funded by the National Strategic Reference Framework (NSRF) 2007-2013: ARCHIMEDES III – Enhancement of research groups in the Technological Education Institutes. The authors are thankful for all this support.
References

1. Aitken, R., Gray, B., Lawson R.: Advertising Effectiveness from a Consumer Perspective. International Journal of Advertising, 27 (2), 279–297 (2008)

2. Amos, C., Holmes, G., Strutton, D.: Exploring the Relationship between Celebrity En-dorser Effects and Advertising Effectiveness. International Journal of Advertising, 27 (2), 209–234 (2008)

3. Blasko, V., Mokwa, M.: Creativity in Advertising: A Janusian Perspective. Journal of Ad-vertising, 15(4), 43–50 (1986)

4. Burke, R., Rangaswamy, A., Wind, J. Eliashberg, J.: A Knowledge-based System for Ad-vertising Design. Marketing Science, 9(3), 212–229 (1990)

5. Chamberlain, J., Poesio, M., Kruschwitz, U.: Phrase Detectives: A Web-based Collaborative Annotation Game. Proceedings of I-Semantics (2008)

6. Chen, Z.: Computational Intelligence for Decision Support. CRC Press, Florida (1999)

7. Ericsson, K., Simon, H.: Protocol Analysis: Verbal Reports as Data (2nd ed.). MIT Press, Boston (1993)

8. Goldenberg, J., Mazursky, D., Solomon, S.: The Fundamental Templates of Quality Ads. Marketing Science, 18 (3), 333–351 (1999)

9. Hill, R., Johnson, L.: Understanding Creative Service: A Qualitative Study of the Advertising Problem Delineation, Communication and Response (apdcr) process. *International Journal of Advertising*, 23(3), 285–308 (2004)

10. M. Lee, H. Shen, J. Z. Huang, and J. S. Marron. Biclustering via sparse singular value decomposition. Biometrics, 66:1087-1095 (2010)

11. MacCrimmon, K., Wagner, C.: Stimulating Ideas Through Creativity Software. Management Science, 40 (11), 1514–1532 (1994)

12. Opas, T.: An Investigation into the Development of a Creativity Support Tool for Advertising. PhD Thesis. Auckland University of Technology (2008)

13. Prensky, M.: Fun, play and games: What makes games engaging? Digital game-based learning, 1-31 (2001)

14. Siorpaes, K., Hepp, M.: Games with a Purpose for the Semantic Web. IEEE Intelligent Systems, 1541-1672 (2008)

15. Vapnik, V.: The Nature of Statistical Learning Theory. Springer-Verlag. ISBN 0-387-98780-0 (1995)

16. von Ahn, L.: Games with a Purpose. IEEE Computer, 39 (6), 92-94 (2006)

17. Walter, A., Nagypal, G.: IMAGENOTION – Collaborative Semantic Annotation of Images and Image Parts and Work Integrated Creation of Ontologies. In: Proceedings of the 1st Conference on Social Semantic Web (CSSW). LNCS, 161-166, Springer (2007)

18. Wang, A., Hoang, C. D. V., Kan, M. Y.: Perspectives on Crowdsourcing Annotations for Natural Language Processing. Technical Report (TRB7/10). The National University of Singapore, School of Computing (2010)

19. Wang, H-C., Cosley, D., Fussell, S. R.: Idea Expander: Supporting Group Brainstorming with Conversationally Triggered Visual Thinking Stimuli. In: Proceedings of the ACM Conference on Computer Supported Cooperative Work (CSCW). Georgia, USA (2010)