Text-to-picture tools, systems, and approaches: a survey

Text-to-picture systems attempt to facilitate high-level, user-friendly communication between humans and computers while promoting understanding of natural language. These systems interpret a natural language text and transform it into a visual format as pictures or images that are either static or dynamic. In this paper, we aim to identify current difficulties and the main problems faced by prior systems, and in particular, we seek to investigate the feasibility of automatic visualization of Arabic story text through multimedia. Hence, we analyzed a number of well-known text-to-picture systems, tools, and approaches. We showed their constituent steps, such as knowledge extraction, mapping, and image layout, as well as their performance and limitations. We also compared these systems based on a set of criteria, mainly natural language processing, natural language understanding, and input/output modalities. Our survey showed that currently emerging techniques in natural language processing tools and computer vision have made promising advances in analyzing general text and understanding images and videos. Furthermore, important remarks and findings have been deduced from these prior works, which would help in developing an effective text-to-picture system for learning and educational purposes.

Related papers:

title: The KidsRoom: A Perceptually-Based Interactive and Immersive Story Environment
abstract: The KidsRoom is a perceptually-based, interactive, narrative playspace for children. Images, music, narration, light, and sound effects are used to transform a normal child's bedroom into a fantasy land where children are guided through a reactive adventure story. The fully automated system was designed with the following goals: (1) to keep the focus of user action and interaction in the physical and not virtual space; (2) to permit multiple, collaborating people to simultaneously engage in an interactive experience combining both real and virtual objects; (3) to use computer-vision algorithms to identify activity in the space without requiring the participants to wear any special clothing or devices; (4) to use narrative to constrain the perceptual recognition, and to use perceptual recognition to allow participants to drive the narrative; and (5) to create a truly immersive and interactive room environment. We believe the KidsRoom is the first multi-person, fully-automated, interactive, narrative environment ever constructed using non-encumbering sensors. This paper describes the KidsRoom, the technology that makes it work, and the issues that were raised during the system's development.1 A demonstration of the project, which complements the material presented here and includes videos, images, and sounds from each part of the story is available at .

title: Animated storytelling system via text
abstract: This paper describes a system, called Interactive e-Hon, for helping children understand difficult content. It works by transforming text into an easily understandable storybook style with animation and dialogue. In this system, easy-to-understand content is created by a semantic tag generator through natural language processing, an animation generator using an animation achieve and animation tables, a dialogue generator using semantic tag information, and a story generator using the Soar AI engine. Through the results of an experiment, this paper describes the advantages of attracting interest, supporting and facilitating understanding, and improving parent-child communication by using Interactive e-Hon.

title: Illustrate It! An Arabic Multimedia Text-to-Picture m-Learning System
abstract: Multimedia learning is the process of building mental representation from words associated with images. Due to the intuitiveness and vividness of visual illustration, many texts to picture systems have been proposed. However, we observe some common limitations in the existing systems, such as the retrieved pictures may not be suitable for educational purposes. Also, finding pedagogic illustrations still requires manual work, which is difficult and time-consuming. The commonly used systems based on the best keyword selection and the best sentence selection may suffer from loss of information. In this paper, we present an Arabic multimedia text-to-picture mobile learning system that is based on conceptual graph matching. Using a knowledge base, a conceptual graph is built from the text accompanied with the pictures in the multimedia repository as well as for the text entered by the user. Based on the matching scores of both conceptual graphs, matched pictures are assigned relative rankings. The proposed system demonstrated its effectiveness in the domain of Arabic stories, however, it can be easily shifted to any educational domain to yield pedagogical illustrations for organizational or institutional needs. Comparisons with the current state-of-the-art systems, based on the best keyword selection and the best sentence selection techniques, have demonstrated significant improvements in the performance. In addition, to facilitate educational needs, conceptual graph visualization and visual illustrative assessment modules are also developed. The conceptual graph visualization enables learners to discover relationships between words, and the visual illustrative assessment allows the system to automatically assess the performance of a learner. The profound user studies demonstrated the efficiency of the proposed multimedia learning system.

title: Pictorial Illustrations Still Improve Students' Learning from Text
abstract: Research conducted primarily during the 1970s and 1980s supported the assertion that carefully constructed text illustrations generally enhance learners' performance on a variety of text-dependent cognitive outcomes. Research conducted throughout the 1990s still strongly supports that assertion. The more recent research has extended pictures-in-text conclusions to alternative media and technological formats and has begun to explore more systematically the “whys,” “whens,” and “for whoms” of picture facilitation, in addition to the “whethers” and “how muchs.” Consideration is given here to both more and less conventional types of textbook illustration, with several “tenets for teachers” provided in relation to each type.

title: Wikify!: linking documents to encyclopedic knowledge
abstract: This paper introduces the use of Wikipedia as a resource for automatic keyword extraction and word sense disambiguation, and shows how this online encyclopedia can be used to achieve state-of-the-art results on both these tasks. The paper also shows how the two methods can be combined into a system able to automatically enrich a text with links to encyclopedic knowledge. Given an input document, the system identifies the important concepts in the text and automatically links these concepts to the corresponding Wikipedia pages. Evaluations of the system show that the automatic annotations are reliable and hardly distinguishable from manual annotations.

title: Applying Wikipedia to Interactive Information Retrieval
abstract: There are many opportunities to improve the interactivity of information retrieval systems beyond the ubiquitous search box. One idea is to use knowledge bases—e.g. controlled vocabularies, classification schemes, thesauri and ontologies—to organize, describe and navigate the information space. These resources are popular in libraries and specialist collections, but have proven too expensive and narrow to be applied to everyday webscale search. Wikipedia has the potential to bring structured knowledge into more widespread use. This online, collaboratively generated encyclopaedia is one of the largest and most consulted reference works in existence. It is broader, deeper and more agile than the knowledge bases put forward to assist retrieval in the past. Rendering this resource machine-readable is a challenging task that has captured the interest of many researchers. Many see it as a key step required to break the knowledge acquisition bottleneck that crippled previous efforts. This thesis claims that the roadblock can be sidestepped: Wikipedia can be applied effectively to open-domain information retrieval with minimal natural language processing or information extraction. The key is to focus on gathering and applying human-readable rather than machine-readable knowledge. To demonstrate this claim, the thesis tackles three separate problems: extracting knowledge from Wikipedia; connecting it to textual documents; and applying it to the retrieval process. First, we demonstrate that a large thesaurus-like structure can be obtained directly from Wikipedia, and that accurate measures of semantic relatedness can be efficiently mined from it. Second, we show that Wikipedia provides the necessary features and training data for existing data mining techniques to accurately detect and disambiguate topics when they are mentioned in plain text. Third, we provide two systems and user studies that demonstrate the utility of the Wikipedia-derived knowledge base for interactive information retrieval.

title: Toward communicating simple sentences using pictorial representations
abstract: This paper addresses and evaluates the hypothesis that pictorial representations can be used to effectively convey simple sentences across language barriers. The paper makes two main contributions. First, it proposes an approach to augmenting dictionaries with illustrative images using volunteer contributions over the Web. The paper describes the PicNet illustrated dictionary, and evaluates the quality and quantity of the contributions collected through several online activities. Second, starting with this illustrated dictionary, the paper describes a system for the automatic construction of pictorial representations for simple sentences. Comparative evaluations show that a considerable amount of understanding can be achieved using visual descriptions of information, with evaluation figures within a comparable range of those obtained with linguistic representations produced by an automatic machine translation system.

title: Enriching textbooks with images
abstract: Textbooks have a direct bearing on the quality of education imparted to the students. Therefore, it is of paramount importance that the educational content of textbooks should provide rich learning experience to the students. Recent studies on understanding learning behavior suggest that the incorporation of digital visual material can greatly enhance learning. However, textbooks used in many developing regions are largely text-oriented and lack good visual material. We propose techniques for finding images from the web that are most relevant for augmenting a section of the textbook, while respecting the constraint that the same image is not repeated in different sections of the same chapter. We devise a rigorous formulation of the image assignment problem and present a polynomial time algorithm for solving the problem optimally. We also present two image mining algorithms that utilize orthogonal signals and hence obtain different sets of relevant images. Finally, we provide an ensembling algorithm for combining the assignments. To empirically evaluate our techniques, we use a corpus of high school textbooks in use in India. Our user study utilizing the Amazon Mechanical Turk platform indicates that the proposed techniques are able to obtain images that can help increase the understanding of the textbook material.

title: Exploring Imaginative Capability and Learning Motivation Difference Through Picture E-Book
abstract: As advanced technology emerges into learning, learning behavior has changed from printed books to e-books with diversified teaching plans, such as picture E-books. Although past research regards picture E-books as successful and effective for children, some studied have reported that the electronic feature may negatively affect children. Briefly, as children are attracted by the games or sounds in electronic books, this paper intends to fill the gap with the augmented reality (AR) technology. The imaginative capability is an important factor to stimulate potential and inspire creativity. The research field of imaginative capability is a critical element of the effect student creativity development in the future; however, to the best of our knowledge, little attention has been focused on imaginative capability with technology, to say nothing of learning in ubiquitous learning environments. To cope with this problem, this paper aims to present a picture E-book based on the AR technology and learning theories to build a learner-centered u-learning environment, and examines how to inspire students’ imaginative capability in three ways: text-based traditional learning, picture-based traditional learning, and picture E-books with the AR technology–in order to determine the differences of students’ learning motivations and imaginative capabilities.

title: Chat with illustration
abstract: Instant messaging service is an important aspect of social media and sprung up in last decades. Traditional instant messaging service transfers information mainly based on textual message, while the visual message is ignored to a great extent. Such instant messaging service is thus far from satisfactory in all-around information communication. In this paper, we propose a novel visual assisted instant messaging scheme named Chat with illustration (CWI), which presents users visual messages associated with textual message automatically. When users start their chat, the system first identifies meaningful keywords from dialogue content and analyzes grammatical and logical relations. Then CWI explores keyword-based image search on a hierarchically clustering image database which is built offline. Finally, according to grammatical and logical relations, CWI assembles these images properly and presents an optimal visual message. With the combination of textual and visual message, users could get a more interesting and vivid communication experience. Especially for different native language speakers, CWI can help them cross language barrier to some degree. In addition, a visual dialogue summarization is also proposed, which help users recall the past dialogue. The in-depth user studies demonstrate the effectiveness of our visual assisted instant messaging scheme.

title: Assistive systems for children with dyslexia and autism
abstract: Autism and dyslexia are both developmental disorders of neural origin. As we still do not understand the neural basis of these disorders fully, technology can take two approaches in helping those affected. The first is to compensate externally for a known difficulty and the other is to achieve the same function using a completely different means. To demonstrate the first option, we are developing a system to compensate for the auditory processing difficulties in case of dyslexia and to demonstrate the second option we propose a system for autism where we remove the need for traditional languages and instead use pictures for communication.

title: Vishit: A Visualizer for Hindi Text
abstract: We outline the design of a visualizer, named Vishit, for texts in the Hindi language. The Hindi language is lingua franca in many states of India where people speak different languages. The visualized text serves as a universal language where seamless communication is needed by many people who speak different languages and have different cultures. Vishit consists of the following three major processing steps: language processing, knowledge base creation and scene generation. Initial results from the Vishit prototype are encouraging.

title: Pictorial Visualization of EMR Summary Interface and Medical Information Extraction of Clinical Notes
abstract: Current Electronic Medical Records (EMR) systems contain large amounts of texts and various tables, to show numerous health data. This type of presentation limits people from promptly determining medical conditions or quickly finding desired information given the large volume of texts that needs to be read. We aim to tackle this as information visualization and extraction problems by creation of easy and intuitive user interfaces for visualizing medical information. We present both a novel graphical interface for visualizing a summary of medical information and an information extraction system that is able to extract and visualize the patient’s medical information from structured clinical notes. The graphical interface allows spatial-position based representations of medical information on human body images (front and back views) and temporal-time based representation of it through interconnected time axes. Medical histories are classified into several event categories and 6 physiological systems to enable efficient browsing of selected information. To extract visual tags from a given clinical note, we use natural language processing. We employ Metamap of 2014AA knowledge source for medical information extraction. We trained 1294 English clinical notes with a Time-Entity Detection model by Apache Open NLP to abstract the time expressions. Extracted location of illness is assigned into one of 6 physiological systems is displayed in spatial interface while the related data are denoted on a horizontal timeline of temporal interface.

title: Visualizing Natural Language Descriptions
abstract: A natural language interface exploits the conceptual simplicity and naturalness of the language to create a high-level user-friendly communication channel between humans and machines. One of the promising applications of such interfaces is generating visual interpretations of semantic content of a given natural language that can be then visualized either as a static scene or a dynamic animation. This survey discusses requirements and challenges of developing such systems and reports 26 graphical systems that exploit natural language interfaces and addresses both artificial intelligence and visualization aspects. This work serves as a frame of reference to researchers and to enable further advances in the field.

title: NewsViz: Emotional Visualization of News Stories
abstract: The NewsViz system aims to enhance news reading experiences by integrating 30 seconds long Flash-animations into news article web pages depicting their content and emotional aspects. NewsViz interprets football match news texts automatically and creates abstract 2D visualizations. The user interface enables animators to further refine the animations. Here, we focus on the emotion extraction component of NewsViz which facilitates subtle background visualization. NewsViz detects moods from news reports. The original text is part-of-speech tagged and adjectives and/or nouns, the word types conveying most emotional meaning, are filtered out and labeled with an emotion and intensity value. Subsequently reoccurring emotions are joined into longer lasting moods and matched with appropriate animation presets. Different linguistic analysis methods were tested on NewsViz: word-by-word, sentence-based and minimum threshold summarization, to find a minimum number of occurrences of an emotion in forming a valid mood. NewsViz proved to be viable for the fixed domain of football news, grasping the overall moods and some more detailed emotions precisely. NewsViz offers an efficient technique to cater for the production of a large number of daily updated news stories. NewsViz bypasses the lack of information for background or environment depiction encountered in similar applications. Further development may refine the detection of emotion shifts through summarization with the full implementation of football and common linguistic knowledge.

title: Bilateral Correspondence Model for Words-and-Pictures Association in Multimedia-Rich Microblogs
abstract: Nowadays, the amount of multimedia contents in microblogs is growing significantly. More than 20% of microblogs link to a picture or video in certain large systems. The rich semantics in microblogs provides an opportunity to endow images with higher-level semantics beyond object labels. However, this raises new challenges for understanding the association between multimodal multimedia contents in multimedia-rich microblogs. Disobeying the fundamental assumptions of traditional annotation, tagging, and retrieval systems, pictures and words in multimedia-rich microblogs are loosely associated and a correspondence between pictures and words cannot be established. To address the aforementioned challenges, we present the first study analyzing and modeling the associations between multimodal contents in microblog streams, aiming to discover multimodal topics from microblogs by establishing correspondences between pictures and words in microblogs. We first use a data-driven approach to analyze the new characteristics of the words, pictures, and their association types in microblogs. We then propose a novel generative model called the Bilateral Correspondence Latent Dirichlet Allocation (BC-LDA) model. Our BC-LDA model can assign flexible associations between pictures and words and is able to not only allow picture-word co-occurrence with bilateral directions, but also single modal association. This flexible association can best fit the data distribution, so that the model can discover various types of joint topics and generate pictures and words with the topics accordingly. We evaluate this model extensively on a large-scale real multimedia-rich microblogs dataset. We demonstrate the advantages of the proposed model in several application scenarios, including image tagging, text illustration, and topic discovery. The experimental results demonstrate that our proposed model can significantly and consistently outperform traditional approaches.

title: Automatic illustration with cross-media retrieval in large-scale collections
abstract: In this paper, we approach the task of finding suitable images to illustrate text, from specific news stories to more generic blog entries. We have developed an automatic illustration system supported by multimedia information retrieval, that analyzes text and presents a list of candidate images to illustrate it. The system was tested on the SAPO-Labs media collection, containing almost two million images with short descriptions, and the MIRFlickr-25000 collection, with photos and user tags from Flickr. Visual content is described by the Joint Composite Descriptor and indexed by a Permutation-Prefix Index. Illustration is a three-stage process using textual search, score filtering and visual clustering. A preliminary evaluation using exhaustive and approximate visual searches demonstrates the capabilities of the visual descriptor and approximate indexing scheme used.

title: Bi-Gram Term Collocations-based Query Expansion Approach for Improving Arabic Information Retrieval
abstract: In the era of information overloading, information retrieval systems are vital applications. Many researchers try to enhance the search results by introducing new methods. Unlike the English language, some languages like Arabic have complex morphological aspects and lack both linguistic and semantic resources. This paper proposes a language-independent semantic-based information retrieval approach, which expands the user query using bi-gram term collocations. The proposed approach has two main contributions. First, the bi-gram term collocations employed to expand the user query are automatically mined from the text corpus, therefore there is no need for an external semantic resource. Second, due to the complexity of the language morphology, the system index is constructed using the corpus words to save the cost and effort of the stemming process. A system prototype for the Arabic language was implemented and evaluated versus the stem-based method. The experimental evaluation has been conducted on the scripts of the Arabic Holy Quran. The evaluation results demonstrate that the proposed system outperforms the stem-based method in terms of precision and recall.

title: Using Unsupervised Deep Learning for Automatic Summarization of Arabic Documents
abstract: Traditional Arabic text summarization (ATS) systems are based on bag-of-words representation, which involve a sparse and high-dimensional input data. Thus, dimensionality reduction is greatly needed to increase the power of features discrimination. In this paper, we present a new method for ATS using variational auto-encoder (VAE) model to learn a feature space from a high-dimensional input data. We explore several input representations such as term frequency (tf), tf-idf and both local and global vocabularies. All sentences are ranked according to the latent representation produced by the VAE. We investigate the impact of using VAE with two summarization approaches, graph-based and query-based approaches. Experiments on two benchmark datasets specifically designed for ATS show that the VAE using tf-idf representation of global vocabularies clearly provides a more discriminative feature space and improves the recall of other models. Experiment results confirm that the proposed method leads to better performance than most of the state-of-the-art extractive summarization approaches for both graph-based and query-based summarization approaches.

title: MADAMIRA: A Fast, Comprehensive Tool for Morphological Analysis and Disambiguation of Arabic
abstract: In this paper, we present MADAMIRA, a system for morphological analysis and disambiguation of Arabic that combines some of the best aspects of two previously commonly used systems for Arabic processing, MADA (Habash and Rambow, 2005; Habash et al., 2009; Habash et al., 2013) and AMIRA (Diab et al., 2007). MADAMIRA improves upon the two systems with a more streamlined Java implementation that is more robust, portable, extensible, and is faster than its ancestors by more than an order of magnitude. We also discuss an online demo (see http://nlp.ldeo.columbia.edu/madamira/) that highlights these aspects.

title: Overview of Text Annotation with Pictures
abstract: The vast array of information available on the Web makes it a challenge for readers to quickly browse through and decide about the importance and relevance of content. Interpreting large-volumes of data is particularly demanding for users with handheld devices in the social media and micro-blogging sphere. Various approaches address this challenge through text summarization, content ranking and personalized recommendation. We describe a family of techniques that help users understand text by automatically annotating text with pictures, referred to as text picturing. The objective is to find a set of pictures that cover the main concepts in a textual snippet. We provide an overview of text picturing, its constituent steps such as knowledge extraction, mapping, scene rendering, as well as application areas. We give a picturing-related literature overview, and list use-cases that offer IT professionals insight into how picturing techniques can be successfully incorporated into real world applications.

title: Automatic Description Generation from Images: A Survey of Models, Datasets, and Evaluation Measures
abstract: Automatic description generation from natural images is a challenging problem that has recently received a large amount of interest from the computer vision and natural language processing communities. In this survey, we classify the existing approaches based on how they conceptualize this problem, viz., models that cast description as either generation problem or as a retrieval problem over a visual or multimodal representational space. We provide a detailed review of existing models, highlighting their advantages and disadvantages. Moreover, we give an overview of the benchmark image datasets and the evaluation measures that have been developed to assess the quality of machine-generated image descriptions. Finally we extrapolate future directions in the area of automatic image description generation.

title: First steps in the investigation of automated text annotation with pictures
abstract: We describe the investigation of automatic annotation of text with pictures, where knowledge extraction uses dependency parsing. Annotation of text with pictures, a form of knowledge visualization, can assist understanding. The problem statement is, given a corpus of images and a short passage of text, extract knowledge (or concepts), and then display that knowledge in pictures along with the text to help with understanding. A proposed solution framework includes a component to extract document concepts, a component to match document concepts with picture metadata, and a component to produce an amalgamated output of text and pictures. A proof-of-concept application based on the proposed framework provides encouraging results

title: Simultaneous image classification and annotation
abstract: Image classification and annotation are important problems in computer vision, but rarely considered together. Intuitively, annotations provide evidence for the class label, and the class label provides evidence for annotations. For example, an image of class highway is more likely annotated with words “road,” “car,” and “traffic” than words “fish,” “boat,” and “scuba.” In this paper, we develop a new probabilistic model for jointly modeling the image, its class label, and its annotations. Our model treats the class label as a global description of the image, and treats annotation terms as local descriptions of parts of the image. Its underlying probabilistic assumptions naturally integrate these two sources of information. We derive an approximate inference and estimation algorithms based on variational methods, as well as efficient approximations for classifying and annotating new images. We examine the performance of our model on two real-world image data sets, illustrating that a single model provides competitive annotation performance, and superior classification performance.

title: Affective image classification via semi-supervised learning from web images
abstract: Affective image classification has drawn increasing research attentions in the affective computing and multimedia communities. Despite many solutions proposed in the literature, it remains a major challenge to bridge the semantic gap between visual features of images and their affective characteristics, partly due to the lack of adequate training samples, which can be largely ascribed to the all-consuming nature of affective image annotation. In this paper, we propose a novel affective image classification algorithm based on semi-supervised learning from web images (SSL-WI). This algorithm consists of four major steps, including color and texture feature extraction, baseline classifier construction, feature selection, and jointly using training images and retrieved web images to re-train the classifier. We have applied this algorithm, the baseline classifier that is not trained by web images, and two state-of-the-art algorithms to differentiating color images in a three-dimensional discrete emotional space. Our results suggest that, with the scheme of semi-supervised learning from web images, the proposed algorithm is able to produce more accurate affective image classification than other three approaches.

title: CNN-feature based automatic image annotation method
abstract: Automatic image annotation(AIA) methods are considered as a kind of efficient schemes to solve the problem of semantic-gap between the original images and their semantic information. However, traditional annotation models work well only with finely crafted manual features. To address this problem, we combined the CNN feature of an image into our proposed model which we referred as SEM by using a famous CNN model-AlexNet. We extracted a CNN feature by removing its final layer and it is proved to be useful in our SEM model. Additionally, based on the experience of the traditional KNN models, we propose a model to address the problem of simultaneously addressing the image tag refinement and assignment while maintaining the simplicity of the KNN model. The proposed model divides the images which have similar features into a semantic neighbor group. Moreover, utilizing a self-defined Bayesian-based model, we distribute the tags which belong to the neighbor group to the test images according to the distance between the test image and the neighbors. At last, the experiments are performed on three typical image datasets corel5k, espGame and laprtc12, which verify the effectiveness of the proposed model.

title: Show and Tell: Lessons Learned from the 2015 MSCOCO Image Captioning Challenge
abstract: Automatically describing the content of an image is a fundamental problem in artificial intelligence that connects computer vision and natural language processing. In this paper, we present a generative model based on a deep recurrent architecture that combines recent advances in computer vision and machine translation and that can be used to generate natural sentences describing an image. The model is trained to maximize the likelihood of the target description sentence given the training image. Experiments on several datasets show the accuracy of the model and the fluency of the language it learns solely from image descriptions. Our model is often quite accurate, which we verify both qualitatively and quantitatively. Finally, given the recent surge of interest in this task, a competition was organized in 2015 using the newly released COCO dataset. We describe and analyze the various improvements we applied to our own baseline and show the resulting performance in the competition, which we won ex-aequo with a team from Microsoft Research.

title: Toward Text-to-Picture Synthesis
abstract: It is estimated that more that 2 million people in the United States have significant communication impairments that result in them relying on methods other than natural speech alone for communication . One type of commonly used augmentative and alternative communication (AAC) system is pictorial communication software such as SymWriter , which uses a lookup table to transliterate each word (or common phrase) in a sentence into an icon. This is an example of converting information between modalities. However, the resulting sequence of icons can be difficult to understand.

title: The Story Picturing Engine---a system for automatic text illustration
abstract: We present an unsupervised approach to automated story picturing. Semantic keywords are extracted from the story, an annotated image database is searched. Thereafter, a novel image ranking scheme automatically determines the importance of each image. Both lexical annotations and visual content play a role in determining the ranks. Annotations are processed using the Wordnet. A mutual reinforcement-based rank is calculated for each image. We have implemented the methods in our Story Picturing Engine (SPE) system. Experiments on large-scale image databases are reported. A user study has been performed and statistical analysis of the results has been presented.

title: The story picturing engine: finding elite images to illustrate a story using mutual reinforcement
abstract: In this paper, we present an approach towards automated story picturing based on mutual reinforcement principle. Story picturing refers to the process of illustrating a story with suitable pictures. In our approach, semantic keywords are extracted from the story text and an annotated image database is searched to form an initial picture pool. Thereafter, a novel image ranking scheme automatically determines the importance of each image. Both lexical annotations and visual content of an image play a role in determining its rank. Annotations are processed using the Wordnet to derive a lexical signature for each image. An integrated region based similarity is also calculated between each pair of images. An overall similarity measure is formed using lexical and visual features. In the end, a mutual reinforcement based rank is calculated for each image using the image similarity matrix. We also present a human behavior model based on a discrete state Markov process which captures the intuition for our technique. Experimental results have demonstrated the effectiveness of our scheme

title: IRM: integrated region matching for image retrieval
abstract: Content-based image retrieval using region segmentation has been an active research area. We present IRM (Integrated Region Matching), a novel similarity measure for region-based image similarity comparison. The targeted image retrieval systems represent an image by a set of regions, roughly corresponding to objects, which are characterized by features reflecting color, texture, shape, and location properties. The IRM measure for evaluating overall similarity between images incorporates properties of all the regions in the images by a region-matching scheme. Compared with retrieval based on individual regions, the overall similarity approach reduces the influence of inaccurate segmentation, helps to clarify the semantics of a particular region, and enables a simple querying interface for region-based image retrieval systems. The IRM has been implemented as a part of our experimental SIMPLIcity image retrieval system. The application to a database of about 200,000 general-purpose images shows exceptional robustness to image alterations such as intensity variation, sharpness variation, color distortions, shape distortions, cropping, shifting, and rotation. Compared with several existing systems, our system in general achieves more accurate retrieval at higher speed.

title: TextRank: Bringing Order into Text
abstract: In this paper, the authors introduce TextRank, a graph-based ranking model for text processing, and show how this model can be successfully used in natural language applications.

title: Easy as ABC? Facilitating Pictorial Communication via Semantically Enhanced Layout
abstract: Pictorial communication systems convert natural language text into pictures to assist people with limited literacy. We define a novel and challenging problem: picture layout optimization. Given an input sentence, we seek the optimal way to lay out word icons such that the resulting picture best conveys the meaning of the input sentence. To this end, we propose a family of intuitive "ABC" layouts, which organize icons in three groups. We formalize layout optimization as a sequence labeling problem, employing conditional random fields as our machine learning method. Enabled by novel applications of semantic role labeling and syntactic parsing, our trained model makes layout predictions that agree well with human annotators. In addition, we conduct a user study to compare our ABC layout versus the standard linear layout. The study shows that our semantically enhanced layout is preferred by non-native speakers, suggesting it has the potential to be useful for people with other forms of limited literacy, too.

title: A text-to-picture system for russian language
abstract: This paper presents motivation and design of the general purpose text-to-picture synthesis system. The described TTP system is designed for Russian language processing and operates with the natural language analysis subsystem, the stage processing subsystem, and the rendering subsystem. Every processing stage has been described and the basic design ideas of the system architecture have been highlighted. User study has been performed and further work reasons are explained.

title: An Ontology Based Approach to Text to Picture Synthesis Systems
abstract: In this paper, we present an ontology based approach to text to picture synthesis. Our approach operates with an ontology in the RDF/XML format. This provides loose coupling of the system components, uni cation of the interacting objects representation and their behaviour, and makes possible veri cation of system information resources.

title: WordsEye: an automatic text-to-scene conversion system
abstract: Natural language is an easy and effective medium for describing visual ideas and mental images. Thus, we foresee the emergence of language-based 3D scene generation systems to let ordinary users quickly create 3D scenes without having to learn special software, acquire artistic skills, or even touch a desktop window-oriented interface. WordsEye is such a system for automatically converting text into representative 3D scenes. WordsEye relies on a large database of 3D models and poses to depict entities and actions. Every 3D model can have associated shape displacements, spatial tags, and functional properties to be used in the depiction process. We describe the linguistic analysis and depiction techniques used by WordsEye along with some general strategies by which more abstract concepts are made depictable.

title: Visual Semantics and Ontology of Eventive Verbs
abstract: Various English verb classifications have been analyzed in terms of their syntactic and semantic properties, and conceptual components, such as syntactic valency, lexical semantics, and semantic/syntactic correlations. Here the visual semantics of verbs, particularly their visual roles, somatotopic effectors, and level-of-detail, is studied. We introduce the notion of visual valency and use it as a primary criterion to recategorize eventive verbs for language visualization (animation) in our intelligent multimodal storytelling system, CONFUCIUS. The visual valency approach is a framework for modelling deeper semantics of verbs. In our ontological system we consider both language and visual modalities since CONFUCIUS is a multimodal system.

title: First steps in the investigation of automated text annotation with pictures
abstract: We describe the investigation of automatic annotation of text with pictures, where knowledge extraction uses dependency parsing. Annotation of text with pictures, a form of knowledge visualization, can assist understanding. The problem statement is, given a corpus of images and a short passage of text, extract knowledge (or concepts), and then display that knowledge in pictures along with the text to help with understanding. A proposed solution framework includes a component to extract document concepts, a component to match document concepts with picture metadata, and a component to produce an amalgamated output of text and pictures. A proof-of-concept application based on the proposed framework provides encouraging results

title: SceneMaker: Automatic Visualisation of Screenplays
abstract: Our proposed software system, SceneMaker, aims to facilitate the production of plays, films or animations by automatically interpreting natural language film scripts and generating multimodal, animated scenes from them. During the generation of the story content, SceneMaker will give particular attention to emotional aspects and their reflection in fluency and manner of actions, body posture, facial expressions, speech, scene composition, timing, lighting, music and camera work. Related literature and software on Natural Language Processing, in particular textual affect sensing, affective embodied agents, visualisation of 3D scenes and digital cinematography are reviewed. In relation to other work, SceneMaker will present a genre-specific text-to-animation methodology which combines all relevant expressive modalities. In conclusion, SceneMaker will enhance the communication of creative ideas providing quick pre-visualisations of scenes.

title: Estimating the visual variety of concepts by referring to Web popularity
abstract: Increasingly sophisticated methods for data processing demand knowledge on the semantic relationship between language and vision. New fields of research like Explainable AI demand to step away from black-boxed approaches and understanding how the underlying semantics of data sets and AI models work. Advancements in Psycholinguistics suggest, that there is a relationship from language perception to how language production and sentence creation work. In this paper, a method to measure the visual variety of concepts is proposed to quantify the semantic gap between vision and language. For this, an image corpus is recomposed using ImageNet and Web data. Web-based metrics for measuring the popularity of sub-concepts are used as a weighting to ensure that the image composition in a dataset is as natural as possible. Using clustering methods, a score describing the visual variety of each concept is determined. A crowd-sourced survey is conducted to create ground-truth values applicable for this research. The evaluations show that the recomposed image corpus largely improves the measured variety compared to previous datasets. The results are promising and give additional knowledge about the relationship of language and vision.