Miracle-dz/UniFAR-FTUs huggingface dataset:
FashionOn: Semantic-guided Image-based Virtual Try-on with Detailed Human and Clothing Information

The image-based virtual try-on system has attracted a lot of research attention. The virtual try-on task is challenging since synthesizing try-on images involves the estimation of 3D transformation from 2D images, which is an ill-posed problem. Therefore, most of the previous virtual try-on systems cannot solve difficult cases, e.g., body occlusions, wrinkles of clothes, and details of the hair. Moreover, the existing systems require the users to upload the image for the target pose, which is not user-friendly. In this paper, we aim to resolve the above challenges by proposing a novel FashionOn network to synthesize user images fitting different clothes in arbitrary poses to provide comprehensive information about how suitable the clothes are. Specifically, given a user image, an in-shop clothing image, and a target pose (can be arbitrarily manipulated by joint points), FashionOn learns to synthesize the try-on images by three important stages: pose-guided parsing translation, segmentation region coloring, and salient region refinement. Extensive experiments demonstrate that FashionOn maintains the details of clothing information (e.g., logo, pleat, lace), as well as resolves the body occlusion problem, and thus achieves the state-of-the-art virtual try-on performance both qualitatively and quantitatively.
ScholarSeq: A Benchmark Dataset for Calculating Sequences of Impact Measures at Individual Level

H-sequence, as the time evolution of h-index, is a promising approach to evaluating a scholar's performance throughout his entire career. However, the lack of benchmark dataset that could be used to compare and evaluate various new and existing h-sequence methods has limited the development of h-sequence or other time series indicators. In order to solve this problem, we have crawled about 7,276,970 papers in computer science field. After that, we find the most cited papers t that could identify out 200 top scientists and 50 ordinary scientists. Finally, we construct a benchmark dataset called ScholarSeq which contains information of 150 particular scholars who are major in computer science field. The dataset includes 37,900 papers published by these authors and 3,263,813 citing papers. ScholarSeq provides citation counts in each individual year for each paper, which can be applied to various academic career impact assessments based on time sequence such as h-sequence. Furthermore, it is of great significance that we package the dataset in paper-time matrices so that informetricians can easily get access to and study various innovative sequences of impact measures. In order to illustrate how to use ScholarSeq, we apply the dataset to analyze 4 state-of-the-arts h-sequence methods. Moreover, we have shared source codes, entire dataset and many other files on our website at http://scholarseq.beyondcloud. cn/.
Relation Understanding in Videos

In this paper, we present our solutions to the grand challenge task "Relation Understanding in Videos" in ACM Multimedia 2019. The challenge task aims to detect instances of target visual relations in a video, where a visual relation instance is represented by a relation triplet with the trajectories of the subject and object. It seems that it is similar to the image relation detection task with the input changed from images to videos. However, video relation detection requires a much more complex pipeline which not only needs to detect objects in each frame, but also track them in the temporal direction. In this challenge, we follow the basic pipeline structure which consists of 3 main separate components: an object detector, an object tracker and a relation predictor. Based on our analysis, there exist data unbalance and label missing problems in the VidOR dataset. We exploit two simple but effective methods to alleviate these problems. We also use trajectory Non-Maximum Suppression and a sliding window method to address the redundancy of trajectory proposals. Experimental results on the challenge task demonstrate that by applying these proposed approaches over the state-of-the-art relation prediction model, the video relation detection performance can be improved with the precision@1 and precision@5 reaching 0.3305 and 0.3507 respectively.
Denoising Method for Microscopic Image of Caenorhabditis Elegans Based on Skeleton-pruning

Aiming at the problem that the traditional segmentation method cannot effectively segment the Caenorhabditis elegans due to the influence of peripheral noise, this paper proposes a denoising method based on skeleton pruning, which uses the skeleton of the image obtained by the early segmentation of Caenorhabditis elegans to determine the position of the coincidence noise. and then the noise is eliminated, thereby improving the accuracy of image segmentation. Experimental results show that the algorithm can effectively improve the accuracy of image segmentation.
Evaluating the GameFlow Model with Different Stakeholders

GameFlow is a widely used model of player enjoyment, with hundreds of applications to designing and evaluating games and game-like experiences since its first publication. Despite its widespread use, there has been limited evaluation of the GameFlow model and previous evaluations have focused on expert reviews from researchers with expertise in player experience. In this paper, we present a study in which 12 different stakeholders (developers, players, researchers, and journalists) conducted expert reviews on a game under development using the GameFlow criteria and provided feedback on the model. We reflect on several key points of improvement for future versions of the GameFlow model and criteria.
Multimodal Dialog System: Generating Responses via Adaptive Decoders

On the shoulders of textual dialog systems, the multimodal ones, recently have engaged increasing attention, especially in the retail domain. Despite the commercial value of multimodal dialog systems, they still suffer from the following challenges: 1) automatically generate the right responses in appropriate medium forms; 2) jointly consider the visual cues and the side information while selecting product images; and 3) guide the response generation with multi-faceted and heterogeneous knowledge. To address the aforementioned issues, we present a Multimodal diAloG system with adaptIve deCoders, MAGIC for short. In particular, MAGIC first judges the response type and the corresponding medium form via understanding the intention of the given multimodal context. Hereafter, it employs adaptive decoders to generate the desired responses: a simple recurrent neural network (RNN) is applied to generating general responses, then a knowledge-aware RNN decoder is designed to encode the multiform domain knowledge to enrich the response, and the multimodal response decoder incorporates an image recommendation model which jointly considers the textual attributes and the visual images via a neural model optimized by the max-margin loss. We comparatively justify MAGIC over a benchmark dataset. Experiment results demonstrate that MAGIC outperforms the existing methods and achieves the state-of-the-art performance.
Document Layout Analysis

Document layout analysis (DLA) is a preprocessing step of document understanding systems. It is responsible for detecting and annotating the physical structure of documents. DLA has several important applications such as document retrieval, content categorization, text recognition, and the like. The objective of DLA is to ease the subsequent analysis/recognition phases by identifying the document-homogeneous blocks and by determining their relationships. The DLA pipeline consists of several phases that could vary among DLA methods, depending on the documents’ layouts and final analysis objectives. In this regard, a universal DLA algorithm that fits all types of document-layouts or that satisfies all analysis objectives has not been developed, yet. In this survey paper, we present a critical study of different document layout analysis techniques. The study highlights the motivational reasons for pursuing DLA and discusses comprehensively the different phases of the DLA algorithms based on a general framework that is formed as an outcome of reviewing the research in the field. The DLA framework consists of preprocessing, layout analysis strategies, post-processing, and performance evaluation phases. Overall, the article delivers an essential baseline for pursuing further research in document layout analysis.
Embodied One-Shot Video Recognition: Learning from Actions of a Virtual Embodied Agent

One-shot learning aims to recognize novel target classes from few examples by transferring knowledge from source classes, under a general assumption that the source and target classes are semantically related but not exactly the same. Based on this assumption, recent work has focused on image-based one-shot learning, while little work has addressed video-based one shot learning. One of the challenges lies in that it is difficult to maintain the disjoint-class assumption for videos, since video clips of target classes may potentially appear in the videos of source classes. To address this issue, we introduce a novel setting, termed as embodied agents based one-shot learning, which leverages synthetic videos produced in a virtual environment to understand realistic videos of target classes. In this setting, we further propose two types of learning tasks: embodied one-shot video domain adaptation and embodied one-shot video transfer recognition. These tasks serve as a testbed for evaluating video related one-shot learning tasks. In addition, we propose a general video segment augmentation method, which significantly facilitates a variety of one-shot learning tasks. Experimental results validate the soundness of our setting and learning tasks, and also show the effectiveness of our augmentation approach to video recognition in the small-sample size regime.
Big Data Service Request Prediction Based on Historical Behavior Time Series

Big data analysis service has been used widely in our society. For example, in financial field, users often use big data analysis services to analyze stocks, assets, and accounts in real-time investment decision-making. Therefore, real-time service response is very important from the perspective of user experience. Caching data and analysis results have been used widely in industrial practice. But these caches generally are passive, rigid and inefficient. Proactive caching approach for time-consuming data services is a worthwhile research problem. In addition, we have encountered this problem in a practical enterprise application. In this paper, we propose a data service request prediction approach based on historical user behavior time series analysis. Results show that this approach can improve the response speed of backend data services effectively.
A Novel Process Mining Algorithm to Discover Non-free Choice Construct from Event Logs

It is always a challenge in the field of process mining to mine the non-free choice construct combining choice and synchronization from the event log. To solve this problem, we propose an improved process mining algorithms based on the genetic process mining. In this paper, we present a new definition of the ordering relations that can determine long distance dependency, which can build the initial population more biasedly, and prepare sufficient high quality individuals for subsequent evolutionary calculations. Then we can reduce the search space and avoid the existence of inferior individual. Mining results are validated by fitness, precision and generalization. Experimental results show that the improved algorithm is better than the existing algorithm for mining non-free choice construct. These improvements make the mining of process model more accurately reflecting the business processes.
Towards Smart Sports Exercises: First Designs

In this paper we introduce the first results from the multidisciplinary Smart Sports Exercises project. Building on a thorough background and the aims of the project, we present our approach and discuss the outcomes of a context analysis and the first design iterations. We propose a model spanning the known 'Augmenting to Transforming' sports dimension, and add a dimension spanning 'Skill-specific to Integrative' exercises, and finish with the description and rationale for six interactive concepts that will form the basis for the next phase of the project.
Neural Storyboard Artist: Visualizing Stories with Coherent Image Sequences

A storyboard is a sequence of images to illustrate a story containing multiple sentences, which has been a key process to create different story products. In this paper, we tackle a new multimedia task of automatic storyboard creation to facilitate this process and inspire human artists. Inspired by the fact that our understanding of languages is based on our past experience, we propose a novel inspire-and-create framework with a story-to-image retriever that selects relevant cinematic images for inspiration and a storyboard creator that further refines and renders images to improve the relevancy and visual consistency. The proposed retriever dynamically employs contextual information in the story with hierarchical attentions and applies dense visual-semantic matching to accurately retrieve and ground images. The creator then employs three rendering steps to increase the flexibility of retrieved images, which include erasing irrelevant regions, unifying styles of images and substituting consistent characters. We carry out extensive experiments on both in-domain and out-of-domain visual story datasets. The proposed model achieves better quantitative performance than the state-of-the-art baselines for storyboard creation. Qualitative visualizations and user studies further verify that our approach can create high-quality storyboards even for stories in the wild.
Understanding the Teaching Styles by an Attention based Multi-task Cross-media Dimensional Modeling

Teaching style plays an influential role in helping students to achieve academic success. In this paper, we explore a new problem of effectively understanding teachers' teaching styles. Specifically, we study 1) how to quantitatively characterize various teachers' teaching styles for various teachers and 2) how to model the subtle relationship between cross-media teaching related data (speech, facial expressions and body motions, content et al.) and teaching styles. Using the adjectives selected from more than 10,000 feedback questionnaires provided by an educational enterprise, a novel concept called Teaching Style Semantic Space (TSSS) is developed based on the pleasure-arousal dimensional theory to describe teaching styles quantitatively and comprehensively. Then a multi-task deep learning based model, Attention-based Multi-path Multi-task Deep Neural Network (AMMDNN), is proposed to accurately and robustly capture the internal correlations between cross-media features and TSSS. Based on the benchmark dataset, we further develop a comprehensive data set including 4,541 full-annotated cross-modality teaching classes. Our experimental results demonstrate that the proposed AMMDNN outperforms (+0.0842% in terms of the concordance correlation coefficient (CCC) on average) baseline methods. To further demonstrate the advantages of the proposed TSSS and our model, several interesting case studies are carried out, such as teaching styles comparison among different teachers and courses, and leveraging the proposed method for teaching quality analysis.
Prediction-CGAN: Human Action Prediction with Conditional Generative Adversarial Networks

The underlying challenge of human action prediction, i.e. maintaining prediction accuracy at very beginning of an action execution, is still not well handled. In this paper, we propose a Prediction Conditional Generative Adversarial Network (Prediction-CGAN) for predicting action, which shares information between completely observed and partially observed videos. Instead of generating future frames, we aim at completing visual representations of unfinished video, which can be directly utilized to predict action label no matter at any progress levels. The Prediction-CGAN incorporates the completion constraint to learn a transformation from incomplete actions to complete actions; the adversarial constraint to ensure the generation has similar discriminative power to complete representation; the label consistency constraint to encourage label consistency between each segment and its corresponding complete video; and the confidence monotonically increasing constraint to yield increasingly accurate predictions as observing more frames. Meanwhile, we introduce a novel adversarial criterion especially for prediction task, which requires the generation is more discriminative than its corresponding incomplete representation, while the generation is less discriminative than its real complete representation. In experiments, we present adequate evaluations to show that the proposed Prediction-CGAN outperforms state-of-the-art methods in action prediction.
A Classification of Fantasy in Serious Games

Through a literature review, we explore fantasy in serious games, the function of fantasy, and what contributes to fantasy. This research firstly regards fantasy as a game characteristic, focusing on the definition of fantasy in games. We find two directions: Mental activities and Artifacts. We construct a taxonomy of fantasy from multiple aspects. This classification can help game researchers and designers understand what fantasy is, and how it can be applied in serious games. This classification enables our next step, to conduct a review of experimental studies, testing whether and how the involvement of fantasy improves the effectiveness of serious games.
Comparing Player Experience in Video Games Played in Virtual Reality or on Desktop Displays: Immersion, Flow, and Positive Emotions

Scientific knowledge of the differences between video games played in virtual reality and on desktop displays in terms of player experience is still limited. Therefore, this study aimed to explore differences in immersion, flow, positive emotions, and psychological needs (i.e., challenge, competence, and tension/annoyance) between a video game played in virtual reality and on a desktop display. Thirty young adults played a racing game in virtual reality (Driveclub VR) and desktop (Driveclub) conditions. The Game Experience Questionnaire was used to assess player experience. The results showed that (a) performance on the game was the same in virtual reality and desktop condition; (b) the video game played in virtual reality was able to elicit more intense positive emotions; (c) the sense of immersion and flow was greater in virtual reality as opposed to the desktop condition; and (d) the fulfillment of psychological needs was independent of the display modality.
Procedural Generation of Video Game Cities for Specific Video Game Genres Using WaveFunctionCollapse (WFC)

Virtual cities as background scenarios can be used for many 3D video game genres like action. However, the procedural generation of virtual cities for specific video game genres is an on-going research problem. In this paper, we seek to establish a grounding for future work into city generation for specific game genres by exploring how game designers approach existing generation tool-sets. Firstly, we look at the video game city Skara Brae from the party-based role-playing game The Bard's Tale and try to replicate it using the Wave Function Collapse (WFC) approach to procedural generation. We show in two experimental conditions which parameters for WFC are suitable for replicating the city. Secondly, a pilot user study with eight users shows how they approach creating different video game cities after they preselect a video game genre. The users' video game level ideas are then discussed, and different output levels are generated using WFC.
AnoPCN: Video Anomaly Detection via Deep Predictive Coding Network

Video anomaly detection is a challenging problem due to the ambiguity and complexity of how anomalies are defined. Recent approaches for this task mainly utilize deep reconstruction methods and deep prediction ones, but their performances suffer when they cannot guarantee either higher reconstruction errors for abnormal events or lower prediction errors for normal events. Inspired by the predictive coding mechanism explaining how brains detect events violating regularities, we address the Anomaly detection problem with a novel deep Predictive Coding Network, termed as AnoPCN, which consists of a Predictive Coding Module (PCM) and an Error Refinement Module (ERM). Specifically, PCM is designed as a convolutional recurrent neural network with feedback connections carrying frame predictions and feedforward connections carrying prediction errors. By using motion information explicitly, PCM yields better prediction results. To further solve the problem of narrow regularity score gaps in deep reconstruction methods, we decompose reconstruction into prediction and refinement, introducing ERM to reconstruct current prediction error and refine the coarse prediction. AnoPCN unifies reconstruction and prediction methods in an end-to-end framework, and it achieves state-of-the-art performance with better prediction results and larger regularity score gaps on three benchmark datasets including ShanghaiTech Campus, CUHK Avenue, and UCSD Ped2.
Mind Band: A Crossmedia AI Music Composing Platform

Various media information in life can have an important impact on our understanding of music. In this paper, we present a demo, Mind Band, which is a Cross-Media artificial intelligent composing platform using our life elements such as emoji, image and humming. In practice, we base our system on the valence-arousal model. We use emotion analysis of life elements to map them to music pieces, which are generated by a Variational Autoencoder - Generative Adversarial Networks model. We provide users with immersive experience by uploading emoji/image/humming and retrieving emotionally related music pieces back. With this platform, everyone can be a composer.
A Survey of Coarse-Grained Reconfigurable Architecture and Design

As general-purpose processors have hit the power wall and chip fabrication cost escalates alarmingly, coarse-grained reconfigurable architectures (CGRAs) are attracting increasing interest from both academia and industry, because they offer the performance and energy efficiency of hardware with the flexibility of software. However, CGRAs are not yet mature in terms of programmability, productivity, and adaptability. This article reviews the architecture and design of CGRAs thoroughly for the purpose of exploiting their full potential. First, a novel multidimensional taxonomy is proposed. Second, major challenges and the corresponding state-of-the-art techniques are surveyed and analyzed. Finally, the future development is discussed.
Walker's Movie Map: Route Vies Synthesis Using Omni-directional Videos

We present a new movie map for walkers that synthesizes street walking views along routes for walkers in an area. We acquired a number of omnidirectional videos from the perspectives of walkers of streets in a certain area (ex. $1km^2$ around Kyoto Station), then perform SLAM to obtain camera poses of key video frames with the coordinates adjusted to the map of the area using reference points. In order to switch one video to another at intersections, we identify the frames of video intersection using camera locations. We refine the intersection frames using visual feature matching. Finally, we synthesize moving route views by switching omnidirectional videos with alignment of the direction of the cameras. The result shows that our method can precisely identify the intersection frames, and generates smooth switching of videos at the intersection.