Miracle-dz/UniFAR-FTUs huggingface dataset:
Multimodal Dialogue Response Generation

Responsing with image has been recognized as an important capability for an intelligent conversational agent. Yet existing works only focus on exploring the multimodal dialogue models which depend on retrieval-based methods, but neglecting generation methods. To fill in the gaps, we first present a new task: multimodal dialogue response generation (MDRG) - given the dialogue history, one model needs to generate a text sequence or an image as response. Learning such a MDRG model often requires multimodal dialogues containing both texts and images which are difficult to obtain. Motivated by the challenge in practice, we consider MDRG under a natural assumption that only limited training examples are available. In such a low-resource setting, we devise a novel conversational agent, Divter, in order to isolate parameters that depend on multimodal dialogues from the entire generation model. By this means, the major part of the model can be learned from a large number of text-only dialogues and text-image pairs respectively, then the whole parameters can be well fitted using the limited training examples. Extensive experiments demonstrate our method achieves state-of-the-art results in both automatic and human evaluation, and can generate informative text and high-resolution image responses.
Information Operations in Turkey: Manufacturing Resilience with Free Twitter Accounts

Following the 2016 US elections Twitter launched their Information Operations (IO) hub where they archive account activity connected to state linked information operations. In June 2020, Twitter took down and released a set of accounts linked to Turkey's ruling political party (AKP). We investigate these accounts in the aftermath of the takedown to explore whether AKP-linked operations are ongoing and to understand the strategies they use to remain resilient to disruption. We collect live accounts that appear to be part of the same network, ~30% of which have been suspended by Twitter since our collection. We create a BERT-based classifier that shows similarity between these two networks, develop a taxonomy to categorize these accounts, find direct sequel accounts between the Turkish takedown and the live accounts, and find evidence that Turkish IO actors deliberately construct their network to withstand large-scale shutdown by utilizing explicit and implicit signals of coordination. We compare our findings from the Turkish operation to Russian and Chinese IO on Twitter and find that Turkey's IO utilizes a unique group structure to remain resilient. Our work highlights the fundamental imbalance between IO actors quickly and easily creating free accounts and the social media platforms spending significant resources on detection and removal, and contributes novel findings about Turkish IO on Twitter.
FacialGAN: Style Transfer and Attribute Manipulation on Synthetic Faces

Facial image manipulation is a generation task where the output face is shifted towards an intended target direction in terms of facial attribute and styles. Recent works have achieved great success in various editing techniques such as style transfer and attribute translation. However, current approaches are either focusing on pure style transfer, or on the translation of predefined sets of attributes with restricted interactivity. To address this issue, we propose FacialGAN, a novel framework enabling simultaneous rich style transfers and interactive facial attributes manipulation. While preserving the identity of a source image, we transfer the diverse styles of a target image to the source image. We then incorporate the geometry information of a segmentation mask to provide a fine-grained manipulation of facial attributes. Finally, a multi-objective learning strategy is introduced to optimize the loss of each specific tasks. Experiments on the CelebA-HQ dataset, with CelebAMask-HQ as semantic mask labels, show our model's capacity in producing visually compelling results in style transfer, attribute manipulation, diversity and face verification. For reproducibility, we provide an interactive open-source tool to perform facial manipulations, and the Pytorch implementation of the model.
A Tactile-enabled Grasping Method for Robotic Fruit Harvesting

In the robotic crop harvesting environment, foreign objects intrusion in the gripper workspace is frequently occurring and unignorable, however, rarely addressed. This paper presents a novel intelligent robotic grasping method capable of handling obstacle interference, which is the first of its kind in the literature. The proposed method combines deep learning algorithms with low-cost tactile sensing hardware on a multi-DoF soft robotic gripper. Through experimental validations, the proposed method demonstrated promising performance in distinguishing various grasping scenarios. The 4-finger independently controlled gripper presented outstanding adaptability to handle various picking scenarios. The overall performance of this work indicated great potential for solving the robotic fruit harvesting challenges.
Electron currents from gradual heating in tilted Dirac cone materials

Materials hosting tilted Dirac/Weyl fermions provide an emergent spacetime structure for the solid state physics. They admit a geometric description in terms of an effective spacetime metric. Using this metric that is rooted in the long-distance behavior of the underlying lattice, we formulate the hydrodynamic theory for tilted Dirac/Weyl materials in 2+12+1 spacetime dimensions. We find that the mingling of space and time through the off-diagonal components of the metric gives rise to: (i) heat and electric currents proportional to the temporal gradient of temperature, \partial_t T∂tT and (ii) a non-zero Hall-like conductance \sigma^{ij}\propto \zeta^i\zeta^jσij∝ζiζj where \zeta^jζj parameterize the tilt in jj’th space direction. The finding (i) above that can be demonstrated in the laboratory, implies that the non-trivial emergent spacetime geometry in these materials empowers them with a fascinating capability to harness naturally available sources of \partial_t T∂tT of hot deserts to produce electric current. We further find a tilt-induced non-Drude contribution to conductivity which can be experimentally disentangled from the usual Drude pole.
Incremental Cross-Domain Adaptation for Robust Retinopathy Screening via Bayesian Deep Learning

Retinopathy represents a group of retinal diseases that, if not treated timely, can cause severe visual impairments or even blindness. Many researchers have developed autonomous systems to recognize retinopathy via fundus and optical coherence tomography (OCT) imagery. However, most of these frameworks employ conventional transfer learning and fine-tuning approaches, requiring a decent amount of well-annotated training data to produce accurate diagnostic performance. This article presents a novel incremental cross-domain adaptation instrument that allows any deep classification model to progressively learn abnormal retinal pathologies in OCT and fundus imagery via few-shot training. Furthermore, unlike its competitors, the proposed instrument is driven via a Bayesian multiobjective function that not only enforces the candidate classification network to retain its prior learned knowledge during incremental training, but also ensures that the network understands the structural and semantic relationships between previously learned pathologies and newly added disease categories to effectively recognize them at the inference stage. The proposed framework, evaluated on six public datasets acquired with three different scanners to screen 13 retinal pathologies, outperforms the state-of-the-art competitors by achieving an overall accuracy and F1 score of 0.9826 and 0.9846, respectively.
Characterizing Improper Input Validation Vulnerabilities of Mobile Crowdsourcing Services

Mobile crowdsourcing services (MCS), enable fast and economical data acquisition at scale and find applications in a variety of domains. Prior work has shown that Foursquare and Waze (a location-based and a navigation MCS) are vulnerable to different kinds of data poisoning attacks. Such attacks can be upsetting and even dangerous especially when they are used to inject improper inputs to mislead users. However, to date, there is no comprehensive study on the extent of improper input validation (IIV) vulnerabilities and the feasibility of their exploits in MCSs across domains. In this work, we leverage the fact that MCS interface with their participants through mobile apps to design tools and new methodologies embodied in an end-to-end feedback-driven analysis framework which we use to study 10 popular and previously unexplored services in five different domains. Using our framework we send tens of thousands of API requests with automatically generated input values to characterize their IIV attack surface. Alarmingly, we found that most of them (8/10) suffer from grave IIV vulnerabilities which allow an adversary to launch data poisoning attacks at scale: 7400 spoofed API requests were successful in faking online posts for robberies, gunshots, and other dangerous incidents, faking fitness activities with supernatural speeds and distances among many others. Lastly, we discuss easy to implement and deploy mitigation strategies which can greatly reduce the IIV attack surface and argue for their use as a necessary complementary measure working toward trustworthy mobile crowdsourcing services.
Algorithms Using Local Graph Features to Predict Epidemics

We study a simple model of epidemics where an infected node transmits the infection to its neighbors independently with probability p. This is also known as the independent cascade or Susceptible-Infected-Recovered (SIR) model with fixed recovery time. The size of an outbreak in this model is closely related to that of the giant connected component in “edge percolation”, where each edge of the graph is kept independently with probability p, studied for a large class of networks including configuration model [Molloy et al. 2011] and preferential attachment [Bollobás and Riordan 2003, Riordan 2005]. Even though these models capture the effects of degree inhomogeneity and the role of super-spreaders in the spread of an epidemic, they only consider graphs that are locally tree like i.e. have a few or no short cycles. Some generalizations of the configuration model were suggested to capture local communities, known as household models [Ball et al. 2009], or hierarchical configuration model [van der Hofstad et al. 2015]. Here, we ask a different question: what information is needed for general networks to predict the size of an outbreak? Is it possible to make predictions by accessing the distribution of small subgraphs (or motifs)? We answer the question in the affirmative for large-set expanders with local weak limits (also known as Benjamini-Schramm limits). In particular, we show that there is an algorithm which gives a (1− ǫ) approximation of the probability and the final size of an outbreak by accessing a constant-size neighborhood of a constant number of nodes chosen uniformly at random. We also present corollaries of the theorem for the preferential attachment model, and study generalizations with household (or motif) structure. The latter was only known for the configuration model.
The Role of Pretrained Representations for the OOD Generalization of RL Agents

Building sample-efficient agents that generalize out-of-distribution (OOD) in real-world settings remains a fundamental unsolved problem on the path towards achieving higher-level cognition. One particularly promising approach is to begin with low-dimensional, pretrained representations of our world, which should facilitate efficient downstream learning and generalization. By training 240 representations and over 10,000 reinforcement learning (RL) policies on a simulated robotic setup, we evaluate to what extent different properties of pretrained VAE-based representations affect the OOD generalization of downstream agents. We observe that many agents are surprisingly robust to realistic distribution shifts, including the challenging sim-to-real case. In addition, we find that the generalization performance of a simple downstream proxy task reliably predicts the generalization performance of our RL agents under a wide range of OOD settings. Such proxy tasks can thus be used to select pretrained representations that will lead to agents that generalize.
Towards Robust Waveform-Based Acoustic Models

We study the problem of learning robust acoustic models in adverse environments, characterized by a significant mismatch between training and test conditions. This problem is of paramount importance for the deployment of speech recognition systems that need to perform well in unseen environments. First, we characterize data augmentation theoretically as an instance of vicinal risk minimization, which aims at improving risk estimates during training by replacing the delta functions that define the empirical density over the input space with an approximation of the marginal population density in the vicinity of the training samples. More specifically, we assume that local neighborhoods centered at training samples can be approximated using a mixture of Gaussians, and demonstrate theoretically that this can incorporate robust inductive bias into the learning process. We then specify the individual mixture components implicitly via data augmentation schemes, designed to address common sources of spurious correlations in acoustic models. To avoid potential confounding effects on robustness due to information loss, which has been associated with standard feature extraction techniques (e.g., fbank and mfcc features), we focus on the waveform-based setting. Our empirical results show that the approach can generalize to unseen noise conditions, with 150% relative improvement in out-of-distribution generalization compared to training using the standard risk minimization principle. Moreover, the results demonstrate competitive performance relative to models learned using a training sample designed to match the acoustic conditions characteristic of test utterances.
Deep learning-based detection of intravenous contrast in computed tomography scans

Purpose: Identifying intravenous (IV) contrast use within CT scans is a key component of data curation for model development and testing. Currently, IV contrast is poorly documented in imaging metadata and necessitates manual correction and annotation by clinician experts, presenting a major barrier to imaging analyses and algorithm deployment. We sought to develop and validate a convolutional neural network (CNN)-based deep learning (DL) platform to identify IV contrast within CT scans. Methods: For model development and evaluation, we used independent datasets of CT scans of head, neck (HN) and lung cancer patients, totaling 133,480 axial 2D scan slices from 1,979 CT scans manually annotated for contrast presence by clinical experts. Five different DL models were adopted and trained in HN training datasets for slice-level contrast detection. Model performances were evaluated on a hold-out set and on an independent validation set from another institution. DL models was then fine-tuned on chest CT data and externally validated on a separate chest CT dataset. Results: Initial DICOM metadata tags for IV contrast were missing or erroneous in 1,496 scans (75.6%). The EfficientNetB4-based model showed the best overall detection performance. For HN scans, AUC was 0.996 in the internal validation set (n = 216) and 1.0 in the external validation set (n = 595). The fine-tuned model on chest CTs yielded an AUC: 1.0 for the internal validation set (n = 53), and AUC: 0.980 for the external validation set (n = 402). Conclusion: The DL model could accurately detect IV contrast in both HN and chest CT scans with near-perfect performance.
Spectral multiplexing of telecom emitters with stable transition frequency

In a quantum network, coherent emitters can be entangled over large distances using photonic channels. In solid-state devices, the required efficient light-emitter interface can be implemented by confining the light in nanophotonic structures. However, fluctuating charges and magnetic moments at the nearby interface then lead to spectral instability of the emitters. Here, we avoid this limitation when enhancing the photon emission up to 70(12)-fold using a Fabry-Perot resonator with an embedded 19-micrometer-thin crystalline membrane, in which we observe around 100 individual erbium emitters. In long-term measurements, they exhibit an exceptional spectral stability of <0.2 megahertz that is limited by the coupling to surrounding nuclear spins. We further implement spectrally multiplexed coherent control and find an optical coherence time of 0.11(1) milliseconds, approaching the lifetime limit of 0.3 milliseconds for the strongest-coupled emitters. Our results constitute an important step toward frequency-multiplexed quantum-network nodes operating directly at a telecommunication wavelength.
Optimum-statistical Collaboration Towards General and Efficient Black-box Optimization

In this paper, we make the key delineation on the roles of resolution and statistical uncertainty in hierarchical bandits-based black-box optimization algorithms, guiding a more general analysis and a more efficient algorithm design. We introduce the \textit{optimum-statistical collaboration}, an algorithm framework of managing the interaction between optimization error flux and statistical error flux evolving in the optimization process. We provide a general analysis of this framework without specifying the forms of statistical error and uncertainty quantifier. Our framework and its analysis, due to their generality, can be applied to a large family of functions and partitions that satisfy different local smoothness assumptions and have different numbers of local optimums, which is much richer than the class of functions studied in prior works. Our framework also inspires us to propose a better measure of the statistical uncertainty and consequently a variance-adaptive algorithm \texttt{VHCT}. In theory, we prove the algorithm enjoys rate-optimal regret bounds under different local smoothness assumptions; in experiments, we show the algorithm outperforms prior efforts in different settings.
Dual Mesh Convolutional Networks for Human Shape Correspondence

Convolutional networks have been extremely successful for regular data structures such as 2D images and 3D voxel grids. The transposition to meshes is, however, not straightforward due to their irregular structure. We explore how the dual, face-based representation of triangular meshes can be leveraged as a data structure for graph convolutional networks. In the dual mesh, each node (face) has a fixed number of neighbors, which makes the networks less susceptible to overfitting on the mesh topology, and also allows the use of input features that are naturally defined over faces, such as surface normals and face areas. We evaluate the dual approach on the shape correspondence task on the Faust human shape dataset and variants of it with different mesh topologies. Our experiments show that results of graph convolutional networks improve when defined over the dual rather than primal mesh. Moreover, our models that explicitly leverage the neighborhood regularity of dual meshes allow improving results further while being more robust to changes in the mesh topology.
Enhanced recovery after surgery (ERAS®) protocol adapted to the Brazilian reality: a prospective cohort study for thoracic patients

Background In Low-Middle Income Countries (LMICs), resource optimization and infrastructure availability are recurrently in debate. In order to assist the development and implementation of guidelines, LMICs often exemplify from High-Income Countries protocols. At the final, it will be: content adaption is often needed. In this study, we demonstrated the preliminary analysis of the Brazilian experience by adapting the ERAS® Protocol for thoracic surgery patients (PROSM). Methods Patients’ data were extracted from the surgical group database that operated in the city of Sao Paulo. Patients’ data were organized for analysis after the institution’s ethics committee gave their approval. Patients’ variables were analyzed and compared to a control group. Subgroup analysis included patients without ICU Admission. Results PROSM patients had reduced ICU length of stay (LOS) (Mean of 0.3±0.58 days, 1.2±1.65 days, P=0.001), Hospital LOS (Mean of 1.6±1.32 days, 3.9±3.25 days, P=0.001) and Chest Drain duration (Median 1.0±1.00 days, 3.0±3.00 days, P=0.001). Analyses of patients that were not admitted to the ICU demonstrated reduced Hospital LOS and Chest drain duration. Cost analysis, such as procedure, daily, and post-surgical costs were also significantly lower towards PROSM group. Conclusions This study revealed important aspects for improvement of the delivered care quality and opportunity for expenditure management. We expect to assist more countries to improve knowledge under the implementation of enhanced protocols.
Association of dietary inflammatory index and metabolic syndrome in the elderly over 55 years in Northern China

Abstract We assessed the association between the dietary inflammatory index (DII) and the development of metabolic syndrome in the elderly over 55 years in Northern China. The data of 1936 Chinese adults aged 55 years and over from a community-based neurological disease cohort study from 2018 to 2019 were analysed. Multiple logistic regression and restricted cubic splines regression were used for analysis, and social demographics, lifestyle and health-related factors were adjusted. In the fully adjusted model, the risk of metabolic syndrome increased by 1·28-fold in people with a pro-inflammatory diet. When we divide the metabolic syndrome by its components, high pro-inflammatory diet and hyperglycaemia, TAG, hypertension and abdominal obesity, we failed to observe a significant association between a high pro-inflammatory diet and HDL-cholesterol. However, these associations are moving in the expected direction. At the same time, the results of BMI subgroup analysis showed that with the increase of DII, obese people are at increased risk of metabolic syndrome, hyperglycaemia, high TAG, hypertension and abdominal obesity. Also in overweight people, the increase in DII is accompanied by an increased risk of hyperglycaemia and abdominal obesity. Higher inflammatory diet is related to metabolic syndrome, hypertension, hyperglycaemia, abdominal obesity and hypertriglyceridaemia. Further research is needed to confirm the role of inflammation and diet in the development of metabolic syndrome; however, it is desirable to reduce the dietary components associated with inflammation.