Reviewerly/exHarmony huggingface dataset:
Where is the Bot in our Team? Toward a Taxonomy of Design Option Combinations for Conversational Agents in Collaborative Work

With rapid progress in machine learning, language technologies and artificial intelligence, conversational agents (CAs) gain rising attention in research and practice as potential non-human teammates, facilitators or experts in collaborative work.However, designers of CAs in collaboration still struggle with a lack of comprehensive understanding of the vast variety of design options in the dynamic field.We address this gap with a taxonomy to help researchers and designers understand the design space and the interrelations of different design options and recognize useful design option combinations for their CAs.We present the iterative development of a taxonomy for the design of CAs grounded in state of the art literature and validated with domain experts.We identify recurring design option combinations and white spots from the classified objects that will inform further research and development efforts.
A Sensor Network Approach for Violence Detection in Smart Cities Using Deep Learning

Citizen safety in modern urban environments is an important aspect of life quality. Implementation of a smart city approach to video surveillance depends heavily on the capability of gathering and processing huge amounts of live urban data. Analyzing data from high bandwidth surveillance video streams provided by large size distributed sensor networks is particularly challenging. We propose here an efficient method for automatic violent behavior detection designed for video sensor networks. Known solutions to real-time violence detection are not suitable for implementation in a resource-constrained environment due to the high processing power requirements. Our algorithm achieves real-time processing on a Raspberry PI-embedded architecture. To ensure separation of temporal and spatial information processing we employ a computationally effective cascaded approach. It consists of a deep neural network followed by a time domain classifier. In contrast with current approaches, the deep neural network input is fed exclusively with motion vector features extracted directly from the MPEG encoded video stream. As proven by results, we achieve state-of-the-art performance, while running on a low computational resources embedded architecture.
A Chaotic Electromagnetic Field Optimization Algorithm Based on Fuzzy Entropy for Multilevel Thresholding Color Image Segmentation

Multilevel thresholding segmentation of color images is an important technology in various applications which has received more attention in recent years. The process of determining the optimal threshold values in the case of traditional methods is time-consuming. In order to mitigate the above problem, meta-heuristic algorithms have been employed in this field for searching the optima during the past few years. In this paper, an effective technique of Electromagnetic Field Optimization (EFO) algorithm based on a fuzzy entropy criterion is proposed, and in addition, a novel chaotic strategy is embedded into EFO to develop a new algorithm named CEFO. To evaluate the robustness of the proposed algorithm, other competitive algorithms such as Artificial Bee Colony (ABC), Bat Algorithm (BA), Wind Driven Optimization (WDO), and Bird Swarm Algorithm (BSA) are compared using fuzzy entropy as the fitness function. Furthermore, the proposed segmentation method is also compared with the most widely used approaches of Otsu’s variance and Kapur’s entropy to verify its segmentation accuracy and efficiency. Experiments are conducted on ten Berkeley benchmark images and the simulation results are presented in terms of peak signal to noise ratio (PSNR), mean structural similarity (MSSIM), feature similarity (FSIM), and computational time (CPU Time) at different threshold levels of 4, 6, 8, and 10 for each test image. A series of experiments can significantly demonstrate the superior performance of the proposed technique, which can deal with multilevel thresholding color image segmentation excellently.
Video-Based Evidence Analysis and Extraction in Digital Forensic Investigation

As a result of the popularity of smart mobile devices and the low cost of surveillance systems, visual data are increasingly being used in digital forensic investigation.Digital videos have been widely used as key evidence sources in evidence identification, analysis, presentation, and report.The main goal of this paper is to develop advanced forensic video analysis techniques to assist the forensic investigation.We first propose a forensic video analysis framework that employs an efficient video/image enhancing algorithm for the low quality of footage analysis.An adaptive video enhancement algorithm based on contrast limited adaptive histogram equalization (CLAHE) is introduced to improve the closed-circuit television (CCTV) footage quality for the use of digital forensic investigation.To assist the video-based forensic analysis, a deeplearning-based object detection and tracking algorithm are proposed that can detect and identify potential suspects and tools from footages.
CycleMatch: A cycle-consistent embedding network for image-text matching

In numerous multimedia and multi-modal tasks from image and video retrieval to zero-shot recognition to multimedia question and answering, bridging image and text representations plays an important and in some cases an indispensable role. To narrow the modality gap between vision and language, prior approaches attempt to discover their correlated semantics in a common feature space. However, these approaches omit the intra-modal semantic consistency when learning the inter-modal correlations. To address this problem, we propose cycle-consistent embeddings in a deep neural network for matching visual and textual representations. Our approach named as CycleMatch can maintain both inter-modal correlations and intra-modal consistency by cascading dual mappings and reconstructed mappings in a cyclic fashion. Moreover, in order to achieve a robust inference, we propose to employ two late-fusion approaches: average fusion and adaptive fusion. Both of them can effectively integrate the matching scores of different embedding features, without increasing the network complexity and training time. In the experiments on cross-modal retrieval, we demonstrate comprehensive results to verify the effectiveness of the proposed approach. Our approach achieves state-of-the-art performance on two well-known multi-modal datasets, Flickr30K and MSCOCO.
Extractive based Text Summarization Using KMeans and TF-IDF

The quantity of information on the internet is massively increasing and gigantic volume of data with numerous compositions accessible openly online become more widespread.It is challenging nowadays for a user to extract the information efficiently and smoothly.As one of the methods to tackle this challenge, text summarization process diminishes the redundant information and retrieves the useful and relevant information from a text document to form a compressed and shorter version which is easy to understand and timesaving while reflecting the main idea of the discussed topic within the document.The approaches of automatic text summarization earn a keen interest within the Text Mining and NLP (Natural Language Processing) communities because it is a laborious job to manually summarize a text document.Mainly there are two types of text summarization, namely extractive based and abstractive based.This paper focuses on the extractive based summarization using K-Means Clustering with TF-IDF (Term Frequency-Inverse Document Frequency) for summarization.The paper also reflects the idea of true K and using that value of K divides the sentences of the input document to present the final summary.Furth more, we have combined the K-means, TF-IDF with the issue of K value and predict the resulting system summary which shows comparatively best results.
Zero-Shot Cross-Lingual Abstractive Sentence Summarization through Teaching Generation and Attention

Abstractive Sentence Summarization (AS-SUM) targets at grasping the core idea of the source sentence and presenting it as the summary.It is extensively studied using statistical models or neural models based on the large-scale monolingual source-summary parallel corpus.But there is no cross-lingual parallel corpus, whose source sentence language is different to the summary language, to directly train a cross-lingual ASSUM system.We propose to solve this zero-shot problem by using resource-rich monolingual AS-SUM system to teach zero-shot cross-lingual ASSUM system on both summary word generation and attention.This teaching process is along with a back-translation process which simulates source-summary pairs.Experiments on cross-lingual ASSUM task show that our proposed method is significantly better than pipeline baselines and previous works, and greatly enhances the cross-lingual performances closer to the monolingual performances.We release the code and data at https://github.com/KelleyYin/Cross-lingual-Summarization.
Collaborative Dialogue in Minecraft

We wish to develop interactive agents that can communicate with humans to collaboratively solve tasks in grounded scenarios.Since computer games allow us to simulate such tasks without the need for physical robots, we define a Minecraft-based collaborative building task in which one player (A, the Architect) is shown a target structure and needs to instruct the other player (B, the Builder) to build this structure.Both players interact via a chat interface.A can observe B but cannot place blocks.We present the Minecraft Dialogue Corpus, a collection of 509 conversations and game logs.As a first step towards our goal of developing fully interactive agents for this task, we consider the subtask of Architect utterance generation, and show how challenging it is.
Generating Summaries with Topic Templates and Structured Convolutional Decoders

Existing neural generation approaches create multi-sentence text as a single sequence.In this paper we propose a structured convolutional decoder that is guided by the content structure of target summaries.We compare our model with existing sequential decoders on three data sets representing different domains.Automatic and human evaluation demonstrate that our summaries have better content coverage.
Fuzzy Region-Based Active Contours Driven by Weighting Global and Local Fitting Energy

Active contour model (ACM) has been a successful method for image segmentation. The existing ACMs poorly segment the images with intensity inhomogeneity or non-homogeneity, and the results highly depend on the initial position of the contour. To overcome these disadvantages, we proposed a fuzzy region-based active contour driven by weighting global and local fitting energy, wherein we propose a fuzzy region energy with local spatial image information, which has been proved convex and ensures the segmentation results independent of initialization, to motivate an initial evolving curve of pseudo level set function (LSF), followed by the pseudo LSF and further smoothed by an edge energy to accurately extract the object boundaries and maintain its distance feature. In addition, in the fuzzy region energy, instead of using the Euler-Lagrange equation to minimize the energy functional, we develop a more direct method to calculate the change of the fuzzy region energy. The experimental results on synthetic and real images with high noise and intensity inhomogeneity show that the proposed model can obtain better performance than the state-of-the-art active contour models, and takes less running time. The code is available at: https://github.com/fangchj2002/FRAGL.
Dual U-Net for the Segmentation of Overlapping Glioma Nuclei

The morphology and surroundings of cells have been routinely used by pathologists to diagnose the pathological subtypes of gliomas and to assess the malignancy of tumors. Thanks to the advent and development of digital pathology technology, it is possible to automatically analyze whole slides of tissue and focus on the nucleus in order to derive a quantitative assessment that can be used for grading, classification, and diagnosis. During the process of computer-assisted diagnosis, the accurate location and segmentation of nuclei from hematoxylin and eosin (H&E)-stained histopathological images is an important step. In this paper, we proposed a U-Net-based multi-task learning network in which the boundary and region information is utilized to improve the segmentation accuracy of glioma nuclei, especially overlapping ones. To refine the segmentation, a classification model is used to predict the boundary, a regression model is used to predict the distance map, and the final segmentation is obtained by using the fusion layers. The proposed approach was compared with other specially designed boundary-aware methods by using a pathological section dataset that consists of 320 glioma cases from the Huashan Hospital at Fudan University. Both the pixel-level and object-level evaluations showed that the structural modification is effective in segmentation with an F1-score of 0.82, a Hausdorff distance (HD) of 3.95, and an aggregated Jaccard index (AJI) of 0.66 (+0.46%, -3.75%, and +4.09% compared with the unimproved methods, respectively). In addition, comparative experiments on multi-organ nuclei segmentation (MoNuSeg) open dataset proved the advanced nature of the proposed method in the field of nuclei segmentation, especially separating touching objects. The proposed method obtains an AJI of 0.59 and an F1-score of 0.79.
Securing Data With Blockchain and AI

Data is the input for various artificial intelligence (AI) algorithms to mine valuable features, yet data in Internet is scattered everywhere and controlled by different stakeholders who cannot believe in each other, and usage of the data in complex cyberspace is difficult to authorize or to validate.As a result, it is very difficult to enable data sharing in cyberspace for the real big data, as well as a real powerful AI.In this paper, we propose the SecNet, an architecture that can enable secure data storing, computing, and sharing in the large-scale Internet environment, aiming at a more secure cyberspace with real big data and thus enhanced AI with plenty of data source, by integrating three key components: 1) blockchain-based data sharing with ownership guarantee, which enables trusted data sharing in the large-scale environment to form real big data; 2) AI-based secure computing platform to produce more intelligent security rules, which helps to construct a more trusted cyberspace; 3) trusted value-exchange mechanism for purchasing security service, providing a way for participants to gain economic rewards when giving out their data or service, which promotes the data sharing and thus achieves better performance of AI.Moreover, we discuss the typical use scenario of SecNet as well as its potentially alternative way to deploy, as well as analyze its effectiveness from the aspect of network security and economic revenue.
LINCE PLUS: Research Software for Behavior Video Analysis

For many years the free LINCE software has been used by many researchers needing a tool to tag behaviors using video recordings, coding behaviors and data register. However, as a research group we envisage new challenges with regards to technology novelties, designing a new tool for the future that can be used in any type of device and closely working on line. LINCE PLUS is a free systematic observational research software that will enable including new trends such as artificial intelligence, web management, collaborative work as well as complex statistical packages, such as integrating the same R language compiler inside the application. It’s time for LINCE PLUS
Educational Services for Intellectual Capital Growth or Transmission of Culture for Transfer of Knowledge—Consumer Satisfaction at St. Petersburg Universities

Higher education has complex roles in society, the economy, and politics; it helps to transmit culture, transfer knowledge, and develop the personality of citizens. This diversity of roles is confronted with the limited resources that are related to the sources of financing, that is, students and their families, the national government, and local authorities, among others. The discussions related to the role of universities concern the economy of knowledge and the digital tools influencing education. The specific case of St. Petersburg universities simultaneously represents the impact of the deep socio-political transition from Soviet society to the liberal principles of a market economy, including the perception of higher education institutions as service sector companies. The services allowed by universities include research and training; however, from the consumer point of view, universities should create specific value: the increase of the intellectual components of human capital. These complex functions are interconnected. During 2017–2018, a survey in St. Petersburg was organised to ascertain the opinion of students, professors, and employers on the quality of education. The results of the survey demonstrate the impact of the exaggerated implementation of the liberal principles on education, both positive and negative. The positive effect is the renewing of content and innovative training techniques due to competition among universities. The negative impacts include the preference for popular disciplines and the opportunistic behavior of students that lose their passion for acquiring knowledge and choose instead the passive attitude of consumers of a competitive service. They are less interested in the sphere of their studies, in searching for a job, in the interaction with other social and economic actors, and even in the research and education options presented by the universities.
DoubleFusion: Real-Time Capture of Human Performances with Inner Body Shapes from a Single Depth Sensor

We propose DoubleFusion, a new real-time system that combines volumetric non-rigid reconstruction with data-driven template fitting to simultaneously reconstruct detailed surface geometry, large non-rigid motion and the optimized human body shape from a single depth camera. One of the key contributions of this method is a double-layer representation consisting of a complete parametric body model inside, and a gradually fused detailed surface outside. A pre-defined node graph on the body parameterizes the non-rigid deformations near the body, and a free-form dynamically changing graph parameterizes the outer surface layer far from the body, which allows more general reconstruction. We further propose a joint motion tracking method based on the double-layer representation to enable robust and fast motion tracking performance. Moreover, the inner parametric body is optimized online and forced to fit inside the outer surface layer as well as the live depth input. Overall, our method enables increasingly denoised, detailed and complete surface reconstructions, fast motion tracking performance and plausible inner body shape reconstruction in real-time. Experiments and comparisons show improved fast motion tracking and loop closure performance on more challenging scenarios. Two extended applications including body measurement and shape retargeting show the potential of our system in terms of practical use.
NTIRE 2019 Challenge on Video Deblurring: Methods and Results

This paper reviews the first NTIRE challenge on video deblurring (restoration of rich details and high frequency components from blurred video frames) with focus on the proposed solutions and results. A new REalistic and Diverse Scenes dataset (REDS) was employed. The challenge was divided into 2 tracks. Track 1 employed dynamic motion blurs while Track 2 had additional MPEG video compression artifacts. Each competition had 109 and 93 registered participants. Total 13 teams competed in the final testing phase. They gauge the state-of-the-art in video deblurring problem.
Concept drift detection and adaptation with hierarchical hypothesis testing

A fundamental issue for statistical classification models in a streaming environment is that the joint distribution between predictor and response variables changes over time (a phenomenon also known as concept drifts), such that their classification performance deteriorates dramatically. In this paper, we first present a hierarchical hypothesis testing (HHT) framework that can detect and also adapt to various concept drift types (e.g., recurrent or irregular, gradual or abrupt), even in the presence of imbalanced data labels. A novel concept drift detector, namely Hierarchical Linear Four Rates (HLFR), is implemented under the HHT framework thereafter. By substituting a widely-acknowledged retraining scheme with an adaptive training strategy, we further demonstrate that the concept drift adaptation capability of HLFR can be significantly boosted. The theoretical analysis on the Type-I and Type-II errors of HLFR is also performed. Experiments on both simulated and real-world datasets illustrate that our methods outperform state-of-the-art methods in terms of detection precision, detection delay as well as the adaptability across different concept drift types.
DeepCU: Integrating both Common and Unique Latent Information for Multimodal Sentiment Analysis

Multimodal sentiment analysis combines information available from visual, textual, and acoustic representations for sentiment prediction. The recent multimodal fusion schemes combine multiple modalities as a tensor and obtain either; the common information by utilizing neural networks, or the unique information by modeling low-rank representation of the tensor. However, both of these information are essential as they render inter-modal and intra-modal relationships of the data. In this research, we first propose a novel deep architecture to extract the common information from the multi-mode representations. Furthermore, we propose unique networks to obtain the modality-specific information that enhances the generalization performance of our multimodal system. Finally, we integrate these two aspects of information via a fusion layer and propose a novel multimodal data fusion architecture, which we call DeepCU (Deep network with both Common and Unique latent information). The proposed DeepCU consolidates the two networks for joint utilization and discovery of all-important latent information. Comprehensive experiments are conducted to demonstrate the effectiveness of utilizing both common and unique information discovered by DeepCU on multiple real-world datasets. The source code of proposed DeepCU is available at https://github.com/sverma88/DeepCU-IJCAI19.
Integrating Knowledge and Reasoning in Image Understanding

Deep learning based data-driven approaches have been successfully applied in various image understanding applications ranging from object recognition, semantic segmentation to visual question answering. However, the lack of knowledge integration as well as higher-level reasoning capabilities with the methods still pose a hindrance. In this work, we present a brief survey of a few representative reasoning mechanisms, knowledge integration methods and their corresponding image understanding applications developed by various groups of researchers, approaching the problem from a variety of angles. Furthermore, we discuss upon key efforts on integrating external knowledge with neural networks. Taking cues from these efforts, we conclude by discussing potential pathways to improve reasoning capabilities.