Reviewerly/exHarmony huggingface dataset:
Context-Aware Image Matting for Simultaneous Foreground and Alpha Estimation

Natural image matting is an important problem in computer vision and graphics. It is an ill-posed problem when only an input image is available without any external information. While the recent deep learning approaches have shown promising results, they only estimate the alpha matte. This paper presents a context-aware natural image matting method for simultaneous foreground and alpha matte estimation. Our method employs two encoder networks to extract essential information for matting. Particularly, we use a matting encoder to learn local features and a context encoder to obtain more global context information. We concatenate the outputs from these two encoders and feed them into decoder networks to simultaneously estimate the foreground and alpha matte. To train this whole deep neural network, we employ both the standard Laplacian loss and the feature loss: the former helps to achieve high numerical performance while the latter leads to more perceptually plausible results. We also report several data augmentation strategies that greatly improve the network's generalization performance. Our qualitative and quantitative experiments show that our method enables high-quality matting for a single natural image.
When Radiology Report Generation Meets Knowledge Graph

Automatic radiology report generation has been an attracting research problem towards computer-aided diagnosis to alleviate the workload of doctors in recent years. Deep learning techniques for natural image captioning are successfully adapted to generating radiology reports. However, radiology image reporting is different from the natural image captioning task in two aspects: 1) the accuracy of positive disease keyword mentions is critical in radiology image reporting in comparison to the equivalent importance of every single word in a natural image caption; 2) the evaluation of reporting quality should focus more on matching the disease keywords and their associated attributes instead of counting the occurrence of N-gram. Based on these concerns, we propose to utilize a pre-constructed graph embedding module (modeled with a graph convolutional neural network) on multiple disease findings to assist the generation of reports in this work. The incorporation of knowledge graph allows for dedicated feature learning for each disease finding and the relationship modeling between them. In addition, we proposed a new evaluation metric for radiology image reporting with the assistance of the same composed graph. Experimental results demonstrate the superior performance of the methods integrated with the proposed graph embedding module on a publicly accessible dataset (IU-RR) of chest radiographs compared with previous approaches using both the conventional evaluation metrics commonly adopted for image captioning and our proposed ones.
Deep Learning for Natural Language Processing in Radiology—Fundamentals and a Systematic Review

Purpose Natural language processing (NLP) enables conversion of free text into structured data. Recent innovations in deep learning technology provide improved NLP performance. We aimed to survey deep learning NLP fundamentals and review radiology-related research. Methods This systematic review was reported according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines. We searched for deep learning NLP radiology studies published up to September 2019. MEDLINE, Scopus, and Google Scholar were used as search databases. Results Ten relevant studies published between 2018 and 2019 were identified. Deep learning models applied for NLP in radiology are convolutional neural networks, recurrent neural networks, long short-term memory networks, and attention networks. Deep learning NLP applications in radiology include flagging of diagnoses such as pulmonary embolisms and fractures, labeling follow-up recommendations, and automatic selection of imaging protocols. Deep learning NLP models perform as well as or better than traditional NLP models. Conclusion Research and use of deep learning NLP in radiology is increasing. Acquaintance with this technology can help prepare radiologists for the coming changes in their field.
Privacy-Preserving Task Recommendation Services for Crowdsourcing

Crowdsourcing is a distributed computing paradigm that utilizes human intelligence or resources from a crowd of workers. Existing solutions of task recommendation in crowdsourcing may leak private and sensitive information about both tasks and workers. To protect privacy, information about tasks and workers should be encrypted before being outsourced to the crowdsourcing platform, which makes the task recommendation a challenging problem. In this paper, we propose a privacy-preserving task recommendation scheme (PPTR) for crowdsourcing, which achieves the task-worker matching while preserving both task privacy and worker privacy. In PPTR, we first exploit the polynomial function to express multiple keywords of task requirements and worker interests. Then, we design a key derivation method based on matrix decomposition, to realize the multi-keyword matching between multiple requesters and multiple workers. Through PPTR, user accountability and user revocation are achieved effectively and efficiently. Extensive privacy analysis and performance evaluation show that PPTR is secure and efficient.
Composite Differential Evolution for Constrained Evolutionary Optimization

When solving constrained optimization problems (COPs) by evolutionary algorithms, the search algorithm plays a crucial role. In general, we expect that the search algorithm has the capability to balance not only diversity and convergence but also constraints and objective function during the evolution. For this purpose, this paper proposes a composite differential evolution (DE) for constrained optimization, which includes three different trial vector generation strategies with distinct advantages. In order to strike a balance between diversity and convergence, one of these three trial vector generation strategies is able to increase diversity, and the other two exhibit the property of convergence. In addition, to accomplish the tradeoff between constraints and objective function, one of the two trial vector generation strategies for convergence is guided by the individual with the least degree of constraint violation in the population, and the other is guided by the individual with the best objective function value in the population. After producing offspring by the proposed composite DE, the feasibility rule and the ε constrained method are combined elaborately for selection in this paper. Moreover, a restart scheme is proposed to help the population jump out of a local optimum in the infeasible region for some extremely complicated COPs. By assembling the above techniques together, a constrained composite DE is proposed. The experiments on two sets of benchmark test functions with various features, i.e., 24 test functions from IEEE CEC2006 and 18 test functions with 10 dimensions and 30 dimensions from IEEE CEC2010, have demonstrated that the proposed method shows better or at least competitive performance against other state-of-the-art methods.
Dynamic Sign Language Recognition Based on Video Sequence With BLSTM-3D Residual Networks

Sign language recognition aims to recognize meaningful movements of hand gestures and is a significant solution in intelligent communication between the deaf community and hearing societies. However, until now, the current dynamic sign language recognition methods still have some drawbacks with difficulties of recognizing complex hand gestures, low recognition accuracy for most dynamic sign language recognition, and potential problems in larger video sequence data training. In order to solve these issues, this paper presents a multimodal dynamic sign language recognition method based on a deep 3-dimensional residual ConvNet and bi-directional LSTM networks, which is named as BLSTM-3D residual network (B3D ResNet). This method consists of three main parts. First, the hand object is localized in the video frames in order to reduce the time and space complexity of network calculation. Then, the B3D ResNet automatically extracts the spatiotemporal features from the video sequences and establishes an intermediate score corresponding to each action in the video sequence after feature analysis. Finally, by classifying the video sequences, the dynamic sign language is accurately identified. The experiment is conducted on test datasets, including DEVISIGN_D dataset and SLR_Dataset. The results show that the proposed method can obtain state-of-the-art recognition accuracy (89.8% on the DEVISIGN_D dataset and 86.9% on SLR_Dataset). In addition, the B3D ResNet can effectively recognize complex hand gestures through larger video sequence data, and obtain high recognition accuracy for 500 vocabularies from Chinese hand sign language.
T-PBFT: An EigenTrust-based practical Byzantine fault tolerance consensus algorithm

Blockchain with these characteristics of decentralized structure, transparent and credible, time-series and immutability, has been considering as a promising technology. Consensus algorithm as one of the core techniques of blockchain directly affects the scalability of blockchain systems. Existing probabilistic finality blockchain consensus algorithms such as PoW, PoS, suffer from power consumptions and low efficiency; while absolute finality blockchain consensus algorithms such as PBFT, HoneyBadgerBFT, could not meet the scalability requirement in a large-scale network. In this paper, we propose a novel optimized practical Byzantine fault tolerance consensus algorithm based on EigenTrust model, namely T-PBFT, which is a multi-stage consensus algorithm. It evaluates node trust by the transactions between nodes so that the high quality of nodes in the network will be selected to construct a consensus group. To reduce the probability of view change, we propose to replace a single primary node with a primary group. By group signature and mutual supervision, we can enhance the robustness of the primary group further. Finally, we analyze T-PBFT and compare it with the other Byzantine fault tolerant consensus algorithms. Theoretical analysis shows that our T-PBFT can optimize the Byzantine fault-tolerant rate, reduce the probability of view change and communication complexity.
Developing Multi-Labelled Corpus of Twitter Short Texts: A Semi-Automatic Method

Facing fast-increasing electronic documents in the Digital Media Age, the need to extract textual features of online texts for better communication is growing. Sentiment classification might be the key method to catch emotions of online communication, and developing corpora with annotation of emotions is the first step to achieving sentiment classification. However, the labour-intensive and costly manual annotation has resulted in the lack of corpora for emotional words. Furthermore, single-label semantic corpora could hardly meet the requirement of modern analysis of complicated user’s emotions, but tagging emotional words with multiple labels is even more difficult than usual. Improvement of the methods of automatic emotion tagging with multiple emotion labels to construct new semantic corpora is urgently needed. Taking Twitter short texts as the case, this study proposes a new semi-automatic method to annotate Internet short texts with multiple labels and form a multi-labelled corpus for further algorithm training. Each sentence is tagged with both the emotional tendency and polarity, and each tweet, which generally contains several sentences, is tagged with the first two major emotional tendencies. The semi-automatic multi-labelled annotation is achieved through the process of selecting the base corpus and emotional tags, data preprocessing, automatic annotation through word matching and weight calculation, and manual correction in case of multiple emotional tendencies are found. The experiments on the Sentiment140 published Twitter corpus demonstrate the effectiveness of the proposed approach and show consistency between the results of semi-automatic annotation and manual annotation. By applying this method, this study summarises the annotation specification and constructs a multi-labelled emotion corpus with 6500 tweets for further algorithm training.
Video Inpainting by Jointly Learning Temporal Structure and Spatial Details

We present a new data-driven video inpainting method for recovering missing regions of video frames. A novel deep learning architecture is proposed which contains two subnetworks: a temporal structure inference network and a spatial detail recovering network. The temporal structure inference network is built upon a 3D fully convolutional architecture: it only learns to complete a low-resolution video volume given the expensive computational cost of 3D convolution. The low resolution result provides temporal guidance to the spatial detail recovering network, which performs imagebased inpainting with a 2D fully convolutional network to produce recovered video frames in their original resolution. Such two-step network design ensures both the spatial quality of each frame and the temporal coherence across frames. Our method jointly trains both sub-networks in an end-to-end manner. We provide qualitative and quantitative evaluation on three datasets, demonstrating that our method outperforms previous learning-based video inpainting methods.
Computer-aided diagnosis system combining FCN and Bi-LSTM model for efficient breast cancer detection from histopathological images

Breast cancer (BC) is one of the most frequent types of cancer that adult females suffer from worldwide. Many BC patients face irreversible conditions and even death due to late diagnosis and treatment. Therefore, early BC diagnosis systems based on pathological breast imagery have been in demand in recent years. In this paper, we introduce an end-to-end model based on fully convolutional network (FCN) and bidirectional long short term memory (Bi-LSTM) for BC detection. FCN is used as an encoder for high-level feature extraction. Output of the FCN is turned to a one-dimensional sequence by the flatten layer and fed into the Bi-LSTM’s input. This method ensures that high-resolution images are used as direct input to the model. We conducted our experiments on the BreaKHis database, which is publicly available at http://web.inf.ufpr.br/vri/breast-cancer-database. In order to evaluate the performance of the proposed method, the accuracy metric was used by considering the five-fold cross-validation technique. Performance of the proposed method was found to be better than previously reported results.
StrongSORT: Make DeepSORT Great Again

Recently, Multi-Object Tracking (MOT) has attracted rising attention, and accordingly, remarkable progresses have been achieved. However, the existing methods tend to use various basic models (e.g, detector and embedding model), and different training or inference tricks, etc. As a result, the construction of a good baseline for a fair comparison is essential. In this paper, a classic tracker, i.e., DeepSORT, is first revisited, and then is significantly improved from multiple perspectives such as object detection, feature embedding, and trajectory association. The proposed tracker, named StrongSORT, contributes a strong and fair baseline for the MOT community. Moreover, two lightweight and plug-and-play algorithms are proposed to address two inherent “missing” problems of MOT: missing association and missing detection. Specifically, unlike most methods, which associate short tracklets into complete trajectories at high computation complexity, we propose an appearance-free link model (AFLink) to perform global association without appearance information, and achieve a good balance between speed and accuracy. Furthermore, we propose a Gaussian-smoothed interpolation (GSI) based on Gaussian process regression to relieve the missing detection. AFLink and GSI can be easily plugged into various trackers with a negligible extra computational cost (1.7 ms and 7.1 ms per image, respectively, on MOT17). Finally, by fusing StrongSORT with AFLink and GSI, the final tracker (StrongSORT++) achieves state-of-the-art results on multiple public benchmarks, i.e., MOT17, MOT20, DanceTrack and KITTI. Codes are available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/dyhBUPT/StrongSORT</uri> and <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/open-mmlab/mmtracking</uri> .
ChatGPT for Language Teaching and Learning

In this technology review, we explore the affordances of the generative AI chatbot ChatGPT for language teaching and learning. In addition to this, we also present debates and drawbacks of ChatGPT. Finally, we present the digital competencies teachers and learners require to use this chatbot ethically and effectively to support language learning.
Non-Local U-Nets for Biomedical Image Segmentation

Deep learning has shown its great promise in various biomedical image segmentation tasks. Existing models are typically based on U-Net and rely on an encoder-decoder architecture with stacked local operators to aggregate long-range information gradually. However, only using the local operators limits the efficiency and effectiveness. In this work, we propose the non-local U-Nets, which are equipped with flexible global aggregation blocks, for biomedical image segmentation. These blocks can be inserted into U-Net as size-preserving processes, as well as down-sampling and up-sampling layers. We perform thorough experiments on the 3D multimodality isointense infant brain MR image segmentation task to evaluate the non-local U-Nets. Results show that our proposed models achieve top performances with fewer parameters and faster computation.
Quad-FPN: A Novel Quad Feature Pyramid Network for SAR Ship Detection

Ship detection from synthetic aperture radar (SAR) imagery is a fundamental and significant marine mission. It plays an important role in marine traffic control, marine fishery management, and marine rescue. Nevertheless, there are still some challenges hindering accuracy improvements of SAR ship detection, e.g., complex background interferences, multi-scale ship feature differences, and indistinctive small ship features. Therefore, to address these problems, a novel quad feature pyramid network (Quad-FPN) is proposed for SAR ship detection in this paper. Quad-FPN consists of four unique FPNs, i.e., a DEformable COnvolutional FPN (DE-CO-FPN), a Content-Aware Feature Reassembly FPN (CA-FR-FPN), a Path Aggregation Space Attention FPN (PA-SA-FPN), and a Balance Scale Global Attention FPN (BS-GA-FPN). To confirm the effectiveness of each FPN, extensive ablation studies are conducted. We conduct experiments on five open SAR ship detection datasets, i.e., SAR ship detection dataset (SSDD), Gaofen-SSDD, Sentinel-SSDD, SAR-Ship-Dataset, and high-resolution SAR images dataset (HRSID). Qualitative and quantitative experimental results jointly reveal Quad-FPN’s optimal SAR ship detection performance compared with the other 12 competitive state-of-the-art convolutional neural network (CNN)-based SAR ship detectors. To confirm the excellent migration application capability of Quad-FPN, the actual ship detection in another two large-scene Sentinel-1 SAR images is conducted. Their satisfactory detection results indicate the practical application value of Quad-FPN in marine surveillance.
POG: Personalized Outfit Generation for Fashion Recommendation at Alibaba iFashion

Increasing demand for fashion recommendation raises a lot of challenges for online shopping platforms and fashion communities. In particular, there exist two requirements for fashion outfit recommendation: the Compatibility of the generated fashion outfits, and the Personalization in the recommendation process. In this paper, we demonstrate these two requirements can be satisfied via building a bridge between outfit generation and recommendation. Through large data analysis, we observe that people have similar tastes in individual items and outfits. Therefore, we propose a Personalized Outfit Generation (POG) model, which connects user preferences regarding individual items and outfits with Transformer architecture. Extensive offline and online experiments provide strong quantitative evidence that our method outperforms alternative methods regarding both compatibility and personalization metrics. Furthermore, we deploy POG on a platform named Dida in Alibaba to generate personalized outfits for the users of the online application iFashion.
Combat COVID-19 with artificial intelligence and big data

To combat COVID-19, at least 29 countries/regions have resorted to digital technology; some embedded it with strict containment measures and achieved great success. We need to improve cryptography and regulations that would enable contact-tracing systems without mass surveillance in order to attain the benefits of location-tracking while protecting individual privacy.
Non-local Meets Global: An Integrated Paradigm for Hyperspectral Image Restoration

Non-local low-rank tensor approximation has been developed as a state-of-the-art method for hyperspectral image (HSI) restoration, which includes the tasks of denoising, compressed HSI reconstruction and inpainting. Unfortunately, while its restoration performance benefits from more spectral bands, its runtime also substantially increases. In this paper, we claim that the HSI lies in a global spectral low-rank subspace, and the spectral subspaces of each full band patch group should lie in this global low-rank subspace. This motivates us to propose a unified paradigm combining the spatial and spectral properties for HSI restoration. The proposed paradigm enjoys performance superiority from the non-local spatial denoising and light computation complexity from the low-rank orthogonal basis exploration. An efficient alternating minimization algorithm with rank adaptation is developed. It is done by first solving a fidelity term-related problem for the update of a latent input image, and then learning a low-dimensional orthogonal basis and the related reduced image from the latent input image. Subsequently, non-local low-rank denoising is developed to refine the reduced image and orthogonal basis iteratively. Finally, the experiments on HSI denoising, compressed reconstruction, and inpainting tasks, with both simulated and real datasets, demonstrate its superiority with respect to state-of-the-art HSI restoration methods.