Reviewerly/exHarmony huggingface dataset:
Multi-modal Multi-instance Learning Using Weakly Correlated Histopathological Images and Tabular Clinical Information

The fusion of heterogeneous medical data is essential in precision medicine to assist medical experts in treatment decision-making. However, there is often little explicit correlation between data from different modalities such as histopathological images and tabular clinical data. Besides, attention-based multi-instance learning (MIL) often lacks sufficient supervision to assign appropriate attention weights for informative image patches and thus generates a good global representation for the whole image. In this paper, we propose a novel multi-modal multi-instance joint learning method, which fuses different modalities and magnification scales as a cross-modal representation to capture the potential complementary information and recalibrate the features in each modality. Furthermore, we leverage the information from tabular clinical data to optimize the MIL bag representation in the imaging modality. The proposed method is evaluated on a challenging medical task, i.e., lymph node metastasis (LNM) prediction of breast cancer, and achieves the state-of-the-art performance with AUC of 0.8844, outperforming the AUC of 0.7111 using histopathological images or the AUC of 0.8312 using tabular clinical data alone. An open-source implementation of our approach can be found at https://github.com/yfzon/Multi-modal-Multi-instance-Learning.
The Impact of the Use of Social Media on Second Language Acquisition

The paper focuses on the use of social media in the process of acquisition of the second language (L2) or foreign language learning (FLL), namely, English in university students. Recently, we have seen a dramatic change in favour of distance learning due to the COVID-19 pandemics, therefore, it could be useful to analyse the impact of social media in FLL as they have been used much more now. It is a study summarizing the most important findings of the most recent research into the impact of social media on L2 acquisition from the psycholinguistic perspective. The paper analyses data from research papers as they appear in Scopus and Web of Science databases in the past ten years (i.e. 2010-2020). The purpose of the study is to describe the current research findings, to analyse their importance, to compare the most important data, and to look for further development of the psycholinguistics of L2. Its primary intention is to look into the acquisition of L2, particularly English as a second language (ESL) in the younger generation, i.e., Generation Z of current university students. It also attempts to highlight both the advantages and disadvantages of using social media in L2 acquisition and therefore to highlight future strategies regarding FLL in university education. The current research in this area provides us with overtly optimistic results in FLL with the assistance of social media, however, the paper claims that the situation of utilizing social media in FLL needs further investigation and analysis, mainly from the psycholinguistic point of view. The current COVID-19 situation makes us aware that the impact of social media will increase due to social distancing.
A novel projector ray-model for 3D measurement in fringe projection profilometry

In fringe projection profilometry (FPP), the geometric calibration of the projector is based mainly on the assumption of the anti-camera model. But that model is not adequate for the non-camera model illumination devices, such as the lens-less laser projector based on MEMS, thus inducing errors in measurement results. Here we present a solution to this problem: A ray-model method to parameterize the distribution of projector illumination in the measurement space instead of determining its intrinsic parameters. Relying on the continuous distribution of phase, we discretize the fringe structured light illumination into a ray set with a variable down-sampling rate according to resolution requirements. On this principle, a flexible calibration strategy was designed to determine each illumination ray with metric spatio-angular parameters. Based on the ray parametric equation, we further derived 3D mapping so that spatial coordinates can be directly mapped from the corresponding truncated image point of the ray. Experimental results demonstrated that the proposed method provided a suitable calibration strategy for FPP systems equipped with a type of non-anti-camera model illumination device. Further, with this technique, errors associated with the projector's luminance nonlinearity are significantly suppressed.
Deep learning from MRI-derived labels enables automatic brain tissue classification on human brain CT

Automatic methods for feature extraction, volumetry, and morphometric analysis in clinical neuroscience typically operate on images obtained with magnetic resonance (MR) imaging equipment. Although CT scans are less expensive to acquire and more widely available than MR scans, their application is currently limited to the visual assessment of brain integrity and the exclusion of co-pathologies. CT has rarely been used for tissue classification because the contrast between grey matter and white matter was considered insufficient. In this study, we propose an automatic method for segmenting grey matter (GM), white matter (WM), cerebrospinal fluid (CSF), and intracranial volume (ICV) from head CT images. A U-Net deep learning model was trained and validated on CT images with MRI-derived segmentation labels. We used data from 744 participants of the Gothenburg H70 Birth Cohort Studies for whom CT and T1-weighted MR images had been acquired on the same day. Our proposed model predicted brain tissue classes accurately from unseen CT images (Dice coefficients of 0.79, 0.82, 0.75, 0.93 and 0.98 for GM, WM, CSF, brain volume and ICV, respectively). To contextualize these results, we generated benchmarks based on established MR-based methods and intentional image degradation. Our findings demonstrate that CT-derived segmentations can be used to delineate and quantify brain tissues, opening new possibilities for the use of CT in clinical practice and research.
Hard-sample Guided Hybrid Contrast Learning for Unsupervised Person Re-Identification

Unsupervised person re-identification (Re-ID) is a promising and very challenging research problem in computer vision. Learning robust and discriminative features with unlabeled data is of central importance to Re-ID. Recently, more attention has been paid to unsupervised Re-ID algorithms based on clustered pseudo-label. However, the previous approaches did not fully exploit information of hard samples, simply using cluster centroid or all instances for contrastive learning. In this paper, we propose a Hard-sample Guided Hybrid Contrast Learning (HHCL) approach combining cluster-level loss with instance-level loss for unsupervised person Re-ID. Our approach applies cluster centroid contrastive loss to ensure that the network is updated in a more stable way. Meanwhile, introduction of a hard instance contrastive loss further mines the discriminative information. Extensive experiments on two popular large-scale Re-ID benchmarks demonstrate that our HHCL outperforms previous state-of-the-art methods and significantly improves the performance of unsupervised person Re-ID. The code of our work is available soon at https://github.com/bupt-ai-cz/HHCL-ReID
DeepPanoContext: Panoramic 3D Scene Understanding with Holistic Scene Context Graph and Relation-based Optimization

Panorama images have a much larger field-of-view thus naturally encode enriched scene context information compared to standard perspective images, which however is not well exploited in the previous scene understanding methods. In this paper, we propose a novel method for panoramic 3D scene understanding which recovers the 3D room layout and the shape, pose, position, and semantic category for each object from a single full-view panorama image. In order to fully utilize the rich context information, we design a novel graph neural network based context model to predict the relationship among objects and room layout, and a differentiable relationship-based optimization module to optimize object arrangement with well-designed objective functions on-the-fly. Realizing the existing data are either with incomplete ground truth or overly-simplified scene, we present a new synthetic dataset with good diversity in room layout and furniture placement, and realistic image quality for total panoramic 3D scene understanding. Experiments demonstrate that our method outperforms existing methods on panoramic scene understanding in terms of both geometry accuracy and object arrangement. Code is available at https://chengzhag.github.io/publication/dpc.
DCR: A Unified Framework for Holistic/Partial Person ReID

Person reidentification (ReID) is a very popular research topic in machine learning and computer vision. According to the occlusions, it can be divided into holistic person ReID and partial person ReID tasks. Occlusions commonly exist in the partial person ReID task but pose or observation perspective changes often occur in the holistic person ReID task; thus, many different algorithms or different network architectures have been designed for each task. However, this approach increases the cost in practice and hinders the development of ReID techniques. To solve this problem, in this work, a unified framework is proposed for holistic/partial person ReID, which can effectively and efficiently address changes in pose or observation perspective and the occlusions in both tasks. In detail, we first employ a fully convolutional network (FCN) to generate feature maps for an arbitrarily sized image and then use spatial pyramid pooling (SPP) to obtain its spatial pyramid feature. Thereafter, to efficiently solve the matching problem between the query image and gallery images, we build a deep spatial pyramid feature collaborative reconstruction model (DCR). In DCR, the reconstruction errors mainly come from similar blocks (uncovered parts), and the influence of the reconstruction errors of dissimilar blocks (covered parts or changed parts) is minimized. In addition, we also use the deep mutual learning approach to jointly learn the features in the training process and promote model training. Experimental results on two partial person ReID datasets and three holistic person ReID datasets demonstrate that the DCR outperforms the state-of-the-art approaches on both tasks and all datasets. Specifically, it outperforms all competitors with a large margin and achieves an improvement of 9.07% and 5.95% over the DSR method (published in CVPR18) on the Partial ReID and Partial-iLIDS datasets with Rank-1, respectively. Similarly, it also achieves an improvement of 5.08% over the VPM method (published in CVPR19) on the DukeMTMC-ReID dataset with Rank-1. Additionally, the running time of our method for each query is more than 7 faster than that of the DSR or DuATM methods.
The role of knowledge creation, absorption and acquisition in determining national competitive advantage

Knowledge has been considered a key factor in countries’ catch-up of competitive advantage. However, few studies have deeply analyzed the complex interaction effect of internal technological capabilities and external knowledge acquisition, and the influence of the institutional environment is often ignored. In this paper, we integrate the knowledge-based view with both the new growth theory and the institution-based view, empirically test the validity of our competitive theoretical framework using the panel data of 29 developing countries and 32 developed countries from 1995 to 2018. The results show that, in countries at different stages of development, internal technological capabilities (knowledge creation capability and knowledge absorption capability) and external knowledge acquisition (external technological knowledge and market information knowledge) have different mechanisms of action, constraints and interaction effects. The findings suggest that countries need to make different catch-up strategies according to their stage of national development.
Unaligned Image-to-Image Translation by Learning to Reweight

Unsupervised image-to-image translation aims at learning the mapping from the source to target domain without using paired images for training. An essential yet restrictive assumption for unsupervised image translation is that the two domains are aligned, e.g., for the selfie2anime task, the anime (selfie) domain must contain only anime (selfie) face images that can be translated to some images in the other domain. Collecting aligned domains can be laborious and needs lots of attention. In this paper, we consider the task of image translation between two unaligned domains, which may arise for various possible reasons. To solve this problem, we propose to select images based on importance reweighting and develop a method to learn the weights and perform translation simultaneously and automatically. We compare the proposed method with state-of-the-art image translation approaches and present qualitative and quantitative results on different tasks with unaligned domains. Extensive empirical evidence demonstrates the usefulness of the proposed problem formulation and the superiority of our method.
Real-time Personal Protective Equipment (PPE) Detection Using YOLOv4 and TensorFlow

The major cause for the continuous spreading of the COVID-19 disease is the human to human transmission of the virus. The virus enters the human body through exposed areas such as the mouth, nose and eyes through touching, coughing, sneezing or close proximity with infected people. So it is imperative people cover specific body parts to avoid contracting the disease and stop the transmission of the virus. Personal Protective Equipment (PPE) such as face mask, face shield, gloves help in this cause. Unfortunately some people are reluctant to wear PPE. In such situations a real-time monitoring system for PPE is needed. In this study we have developed a detector which can detect in real time if people are wearing PPE or not. The detector is developed using YOLOv4 computer vision model which specially performs well in real time object detection. We have prepared a combined dataset consisting of collected images and our captured images. We have annotated our images and image augmentation is done on all the training set images. We have also converted the detector weight to TensorFlow format to check live detection performance and added features like live object count and record keeping. In this paper we propose a method using YOLOv4 model with combined dataset and other techniques to detect four classes of objects. We get promising results from the tests and mAP of the object detector is 79%.
Direct adaptive control for nonlinear systems using a TSK fuzzy echo state network based on fractional-order learning algorithm

This paper presents a new Takagi-Sugeno-Kang fuzzy Echo State Neural Network (TSKFESN) structure to design a direct adaptive control for uncertain SISO nonlinear systems. The proposed TSKFESN structure is based on the echo state neural network framework containing multiple sub-reservoirs. Each sub-reservoir is weighted with a TSK fuzzy rule. The adaptive law of the TSKFESN-based direct adaptive controller is derived by using a fractional-order sliding mode learning algorithm. Moreover, the Lyapunov stability criterion is employed to verify the convergence of the fractional-order adaptive law of the controller parameters. The evaluation of the proposed direct adaptive control scheme is verified using two case studies, the regulation problem of a torsional pendulum and the speed control of a direct current (DC) machine as a real-time application. The simulation and the experimental results show the effectiveness of the proposed control scheme.
Turn-to-Diarize: Online Speaker Diarization Constrained by Transformer Transducer Speaker Turn Detection

In this paper, we present a novel speaker diarization system for streaming on-device applications. In this system, we use a transformer transducer to detect the speaker turns, represent each speaker turn by a speaker embedding, then cluster these embeddings with constraints from the detected speaker turns. Compared with conventional clustering-based diarization systems, our system largely reduces the computational cost of clustering due to the sparsity of speaker turns. Unlike other supervised speaker diarization systems which require annotations of time-stamped speaker labels for training, our system only requires including speaker turn tokens during the transcribing process, which largely reduces the human efforts involved in data collection.
Motion-Aware Dynamic Architecture for Efficient Frame Interpolation

Video frame interpolation aims to synthesize accurate intermediate frames given a low-frame-rate video. While the quality of the generated frames is increasingly getting better, state-of-the-art models have become more and more computationally expensive. However, local regions with small or no motion can be easily interpolated with simple models and do not require such heavy compute, whereas some regions may not be correct even after inference through a large model. Thus, we propose an effective framework that assigns varying amounts of computation for different regions. Our dynamic architecture first calculates the approximate motion magnitude to use as a proxy for the difficulty levels for each region, and decides the depth of the model and the scale of the input. Experimental results show that static regions pass through a smaller number of layers, while the regions with larger motion are downscaled for better motion reasoning. In doing so, we demonstrate that the proposed framework can significantly reduce the computation cost (FLOPs) while maintaining the performance, often up to 50% when interpolating a 2K resolution video.
Parallel Rectangle Flip Attack: A Query-based Black-box Attack against Object Detection

Object detection has been widely used in many safety- critical tasks, such as autonomous driving. However, its vulnerability to adversarial examples has not been sufficiently studied, especially under the practical scenario of black-box attacks, where the attacker can only access the query feedback of predicted bounding-boxes and top- 1 scores returned by the attacked model. Compared with black-box attack to image classification, there are two main challenges in black-box attack to detection. Firstly, even if one bounding-box is successfully attacked, another sub- optimal bounding-box may be detected near the attacked bounding-box. Secondly, there are multiple bounding- boxes, leading to very high attack cost. To address these challenges, we propose a Parallel Rectangle Flip Attack (PRFA) via random search. We explain the difference between our method with other attacks in Fig. 1. Specifically, we generate perturbations in each rectangle patch to avoid sub-optimal detection near the attacked region. Besides, utilizing the observation that adversarial perturbations mainly locate around objects’ contours and critical points under white-box attacks, the search space of attacked rectangles is reduced to improve the attack efficiency. Moreover, we develop a parallel mechanism of attacking multiple rectangles simultaneously to further accelerate the attack process. Extensive experiments demonstrate that our method can effectively and efficiently attack various popular object detectors, including anchor-based and anchor- free, and generate transferable adversarial examples.
Adversarial steganography based on sparse cover enhancement

CNN (Convolutional Neural Network) steganalyzers achieve enormous improvements in detecting stego images. However, they are easily deceived by adversarial steganography, which combines adversarial attack and steganography. Currently, there are two kinds of adversarial steganography, function separation and cover enhancement. ADV-EMB (ADVersarial EMBedding) is a typical function separation method. It forces the steganographic modifications along side the gradient directions of the target CNN steganalyzer on partial image elements. It results in relatively low deceiving success rate against the target model. ADS (ADversarial Steganography) is the first adversarial steganography, which is based on cover enhancement. It introduces much distortions, so it can be easily detected by non-target steganalyzers. To overcome such defects of the previous works, in this paper, we propose a novel cover enhancement method, denoted as SPS-ENH (SParSe ENHancement). Through sparse ±1 adversarial perturbations, we effectively compress the distortions caused in cover enhancement. In addition, a re-trying scheme is introduced to further reduce the distortion scale. Extensive experiments show that the proposed method outperforms the previous works in the average classification error rates under non-target steganalyzers and deceiving success rates against target CNN models. When combining with the min–max strategy, the proposed method converges in less iterations and provides higher security level than ADV-EMB.
Rethinking the Ranks of Visual Channels

Data can be visually represented using visual channels like position, length or luminance. An existing ranking of these visual channels is based on how accurately participants could report the ratio between two depicted values. There is an assumption that this ranking should hold for different tasks and for different numbers of marks. However, there is surprisingly little existing work that tests this assumption, especially given that visually computing ratios is relatively unimportant in real-world visualizations, compared to seeing, remembering, and comparing trends and motifs, across displays that almost universally depict more than two values. To simulate the information extracted from a glance at a visualization, we instead asked participants to immediately reproduce a set of values from memory after they were shown the visualization. These values could be shown in a bar graph (position (bar)), line graph (position (line)), heat map (luminance), bubble chart (area), misaligned bar graph (length), or 'wind map' (angle). With a Bayesian multilevel modeling approach, we show how the rank positions of visual channels shift across different numbers of marks (2, 4 or 8) and for bias, precision, and error measures. The ranking did not hold, even for reproductions of only 2 marks, and the new probabilistic ranking was highly inconsistent for reproductions of different numbers of marks. Other factors besides channel choice had an order of magnitude more influence on performance, such as the number of values in the series (e.g., more marks led to larger errors), or the value of each mark (e.g., small values were systematically overestimated). Every visual channel was worse for displays with 8 marks than 4, consistent with established limits on visual memory. These results point to the need for a body of empirical studies that move beyond two-value ratio judgments as a baseline for reliably ranking the quality of a visual channel, including testing new tasks (detection of trends or motifs), timescales (immediate computation, or later comparison), and the number of values (from a handful, to thousands).