Miracle-dz/UniFAR-FTUs huggingface dataset:
Parameter Efficient Fine-tuning for Multi-task Learning

This project aims to implement, utilize, and improve upon the BERT model to perform sentiment analysis and other downstream tasks. In the first part of our project, we fully implement the original BERT model and test it on sentiment analysis – in the second part, we fine-tune and extend the model for optimal performance on paraphrase detection and semantic textual similarity. To do so, we implement Projected Attention Layers (PALs) Stickland and Murray (2019), adapters Houlsby et al. (2019), and prefix tuning Li and Liang (2021) to achieve optimal performance over multi-tasks while being efficient. We also experiment with changes to the BERT model architecture by implementing Sentence-BERT Reimers and Gurevych (2019) and modifying the downstream classifier head architecture. We experiment with different model architectures, adaptation modules, samplers, adding additional training data, and hyper-parameter configurations. We find Sentence-BERT learns more semantically meaningful sentence embeddings and has better performance on the paraphrase and similarity tasks. PAL, prefix, and adapter improve average performance by about 8% when pretraining, and have comparable performance to full fine-tuning with less than 10% of trainable parameters.
Fine Tuning Multi Downstream Tasks based on BERT with Gradient Surgery

Bidirectional Encoder Representations from Transformers (BERT) is a transformer-based model that generates contextual word representations. Such representation could be utilized to multiple downstream tasks, including sentiment analysis, paraphrase detection and semantic textual similarity analysis. In order to create more robust and semantically-rich sentence embeddings, it is important to fine-tune the BERT weights with task-relative data efficiently. In the project, we have tested different fine-tuning techniques for improving the model performance on the aforementioned three downstream tasks. These tests include changes in the architecture (dense layer size, Siamese network), in the optimization step (train separately or together, whether to apply gradient surgery or not), and in the training setting (whether to reshuffle data after each epoch, batch size, regularization). After comparison between different tests, under the limit of AWS g5.2xlarge instance, the best performance is achieved with (1) Add Siamese network and cosine similarity for semantic textual similarity task (2) Pass embeddings for two sentences with their difference into the dense linear layer for the paraphrase task (3) Multi-task training with gradient surgery (4) Reshuffle dataloader after each training epoch (5) Apply l2 regularization (6) Train under batch size 2 for sentiment and similarity task, and batch size 32 for paraphrase task. The model achieves 52.1% accuracy for sentiment task, 82.8% accuracy for paraphrase task and 0.820 Pearson Correlation for semantic textual similarity task on the dev set, and 52.4%, 82.7% and 0.792 on the test set.
CONSTRUCTING LANGUAGES TO EXPLORE THEORETICAL PRINCIPLES

The construction of languages has always been related to linguistics. It addresses linguistic notions or relevant questions from a non-academic point of view. In this paper, we propose a method inspired by experimental archaeology. The experiment consists in trying to obtain an artefact similar to the one observed using this or that construction method. An equivalent approach in linguistics would be the generation of linguistic systems based on explicitly formulated principles. Trying to generate similar systems pushes the linguist to explicitly define the principles that are needed and to explore all their consequences. In this context, we show that the use of notions induced by the observation of natural languages leads to a certain degree of circularity and that it is therefore more interesting to explore a priori principles based on very general assumptions.
MultitaskBERT with Contrastive Pretraining and Fine-Grained Feature Learning

Multitask learning has gained considerable attention as an effective approach for improving data efficiency and reducing training overhead when addressing multiple natural language processing (NLP) tasks. Nonetheless, it presents unique optimization challenges, such as maintaining embedding space quality and reconciling contradictory gradient descent directions, which substantially influence overall performance. To tackle these challenges, we introduce an innovative Multitask BERT model that integrates pre-training and fine-tuning phases by combining contrastive learning methodologies, multitask learning strategies, fine-grained embedding representations, and regularized multitask optimization techniques. Our model’s performance is assessed on three diverse downstream NLP tasks, namely Semantic Textual Similarity (STS), Sentiment Analysis, and Paraphrase Detection. Empirical results from our experiments indicate that incorporating contrastive learning significantly enhances the consistency of the embedding space, thereby augmenting BERT’s expressive capabilities. Additionally, we propose regularized multitask optimization methods tailored for various downstream tasks, designed to amalgamate shared information while mitigating information conflicts. This fusion not only expedites the training process but also bolsters prediction accuracy and optimization stability.By integrating these advanced techniques, our novel multitask BERT model demonstrates considerable performance gains in comparison to established baseline models, underscoring its efficacy for a range of NLP applications.
Automated Basketball Video Captioning

In this paper, we investigate automated machine captioning of basketball videos. We expand on the paper Sports Video Analysis on Large Scale Data [1], which provides a novel NBA video captioning dataset, and a unified approach for video captioning, the SportsFormer model. We identify a main shortcoming of the paper’s approach, which is that it uses a very complex pipeline that involves basketball specific vision models. We demonstrate that with additional data gathering, modifications to the decoder architecture and a multi-task learning framework, we can have a simpler and high performing model composed of a pre-trained video trans-former, a pre-trained encoder to process the video encodings, and a decoder model. This approach eliminates the use of three object detection models, a segmentation model, a vision transformer, and two other transformers from the SportsFormer architecture. Our method removes the need for training basketball specific vision models, which potentially makes this framework more adaptable to other sports.
Predicting Emergency Department Disposition from Radiology Reports

In this project, we propose analyzing emergency department disposition through the lens of the radiology reports. Specifically, we have two objectives, conduct an unsupervised multi-label classification to extract medical conditions, followed by supervised classification to predict patients’ disposition. Several unsupervised label extraction methods were used to obtain medical conditions by considering CheXbert labels as a ground truth. We evaluate methods for supervised classification of the disposition by combining the extracted medical conditions and the radiology reports. Our work demonstrates superior performance of using a pretrained BART-MNLI model for zero-shot label extraction. We find that models trained using the extracted medical conditions together with medical annotations outperforms the other methods to predict patients’ disposition.
Quantifiable integrity for Linked Data on the web

We present an approach to publish Linked Data on the Web with quantifiable integrity using Web technologies, and in which rational agents are incentivised to contribute to the integrity of the link network. To this end, we introduce self-verifying resource representations, that include Linked Data Signatures whose signature value is used as a suffix in the resource’s URI. Links among such representations, typically managed as web documents, contribute therefore to preserving the integrity of the resulting document graphs. To quantify how well a document’s integrity can be relied on, we introduce the notion of trust scores and present an interpretation based on hubs and authorities. In addition, we present how specific agent behaviour may be induced by the choice of trust score regarding their optimisation, e.g., in general but also using a heuristic strategy called Additional Reach Strategy (ARS). We discuss our approach in a three-fold evaluation: First, we evaluate the effect of different graph metrics as trust scores on induced agent behaviour and resulting evolution of the document graph. We show that trust scores based on hubs and authorities induce agent behaviour that contributes to integrity preservation in the document graph. Next, we evaluate different heuristics for agents to optimise trust scores when general optimisation strategies are not applicable. We show that ARS outperforms other potential optimisation strategies. Last, we evaluate the whole approach by examining the resilience of integrity preservation in a document graph when resources are deleted. To this end, we propose a simulation system based on the Watts–Strogatz model for simulating a social network. We show that our approach produces a document graph that can recover from such attacks or failures in the document graph.
A denoising method based on combined Curvelet and Wavelet transform

Summary This paper is based on curvelet transform and wavelet transform. We propose a denoising method based on curvelet and wavelet transform to solve the problem that when we use curvelet transform there remains background noise and generate false signal. To the noised seismogram, select appropriate threshold, denoise the data followed by the use of curvelet transform and wavelet transform. The result shows that curvelet-wavelet transform can retain the valid signal while remove the background noise and false signal, and get a high SNR seismic data.
Estimating from a More General Time-Series Cum Cross-Section Data Structure

This note deals with the problem of parameter estimation from combined time-series-cross-sec tion data. A few works exist on the subject, e.g., Kuh [7], Mundlak [9], [10], Balestra and Nerlove [2], Hoch [6], Hildreth [5], Wallace and Hussain [13]. 1 Some of these [2], [5], [10], [13], used structures identical to those of error component models. In [13], the error was assumed to consist of three independent components? one associated with time, another with cross-sectional units and the other with the interaction between time and cross-sectional units. In this note we generalize that structure to include four independent com ponents consisting, ceteris paribus, of two, rather than one, cross-sectional components. Examples of one component (of these two) may be countries or regions, of the other may be areas within the coun tries or regions, being especially meaningful in international or interregional studies. For instance, one may be interested in studies of the contributions of multinational corporations in the regional and area developments of the world through their investment participation. Or, in the national con text, one may be interested, for example, in the variation of the parameters of the consumption function over the different regions of a country as well as provinces or smaller divisions of a region. In the Canadian case, differences are known to exist in the consumption pattern of people of Ontario and the Maritime provinces as for two of the regions and between the North and the South of Ontario as for two of the areas within a region. As in the three (error-) component model, the last component in the four component model is an interaction between time and all the cross-section units. But now this is to be understood as a com posite, rather than a unique, entity. It implicitly includes two first order interactions, namely, those between time and either of the cross-section dimen sions and one second order interaction, that is, that between time and the first and the second cross section dimension. On the whole, the composite interaction component as well as the other com ponents (due to time and cross-sections) of the error variable are random. We assume however, unlike some studies of the past, that time and each of the cross-section variables give rise to some fixed effects, knowledge of which might be important in certain contexts. With this introduction, we go on to lay bare (Section II) the model, then estimate (Section III) its parameters on the basis of Aitken's Generalized Estimators and Covariance Estimators, the assump tion being that the components of the error variance are known. When these are not known, we use alternative methods. At the end (Section IV) we conclude by comparing briefly the various esti mators.
A NEW METHOD UNFOLDING SPECTRA FOR THE SOFT X-RAY SPECTROMETER

A new method unfolding soft X-ray spectra for a soft X-ray spectrometer(SXS),which is based on maximum entropy principle(ME),is put forward.The spline fit method is used to smooth the discontinuities between the channels at the filters edge responses.The convergent speed of ME is rapid,the stability and the deconvolution accuracy appear to be good.ME's unfolding error can achieve 5%. The method is capable of unfolding the soft X-ray spetra for the soft X-ray spectrometer. [
M ACHINE L EARNING A LGORITHMS FOR C REDIT C ARD F RAUD D ETECTION

Fraud is a critical issue in our society today. Losses due to payment fraud are on the increase as e-commerce keeps evolving. Organizations, governments, and individuals have experienced huge losses due to payment. Merchant Savvy projects that global losses due to payment fraud will increase to about $40.62 billion in 2027 . Among all payment fraud, credit card fraud results in a higher loss. Therefore, we intend to leverage the potential of machine learning to deal with the problem of fraud in credit cards which can be generalized to other fraud types. This paper compares the performance of logistic regression, decision trees, random forest classifier, isolation forest, local outlier factor, and one-class support vector machines (SVM) based on their AUC and F1-score. We applied a smote technique to handle the imbalanced nature of the data and compared the performance of the supervised models on the oversampled data to the raw data. From the results, the Random Forest classifier outperformed the other models with a higher AUC score and better f1-score on both the actual and oversampled data. Oversampling the data didn't change the result of the decision trees. One-class SVM performs better than isolation forest in terms of AUC score but has a very low f1-score compared to isolation forest. The local outlier factor had the poorest performance.
Investigating Traffic Analysis Attacks on Apple iCloud Private Relay

The iCloud Private Relay (PR) is a new feature introduced by Apple in June 2021 that aims to enhance online privacy by protecting a subset of web traffic from both local eavesdroppers and websites that use IP-based tracking. The service is integrated into Apple’s latest operating systems and uses a two-hop architecture where a user’s web traffic is relayed through two proxies run by disjoint entities. PR’s multi-hop architecture resembles traditional anonymity systems such as Tor and mix networks. Such systems, however, are known to be susceptible to a vulnerability known as traffic analysis: an intercepting adversary (e.g., a malicious router) can attempt to compromise the privacy promises of such systems by analyzing characteristics (e.g., packet timings and sizes) of their network traffic. In particular, previous works have widely studied the susceptibility of Tor to website fingerprinting and flow correlation, two major forms of traffic analysis. In this work, we are the first to investigate the threat of traffic analysis against the recently introduced PR. First, we explore PR’s current architecture to establish a comprehensive threat model of traffic analysis attacks against PR. Second, we quantify the potential likelihood of these attacks against PR by evaluating the risks imposed by real-world AS-level adversaries through empirical measurement of Internet routes. Our evaluations show that some autonomous systems are in a particularly strong position to perform traffic analysis on a large fraction of PR traffic. Finally, having demonstrated the potential for these attacks to occur, we evaluate the performance of several flow correlation and website fingerprinting attacks over PR traffic. Our evaluations show that PR is highly vulnerable to state-of-the-art website fingerprinting and flow correlation attacks, with both attacks achieving high success rates. We hope that our study will shed light on the significance of traffic analysis to the current PR deployment, convincing Apple to perform design adjustments to alleviate the risks.
Eos: Efficient Private Delegation of zkSNARK Provers

Succinct zero knowledge proofs (i.e. zkSNARKs) are powerful cryptographic tools that enable a prover to convince a verifier that a given statement is true without revealing any additional information. Their attractive privacy properties have led to much academic and industrial interest. Unfortunately, existing systems for generating zkSNARKs are expensive, which limits the applications in which these proofs can be used. One approach is to take advantage of powerful cloud servers to generate the proof. However, existing techniques for this (e.g., DIZK) sacrifice privacy by revealing secret information to the cloud machines. This is problematic for many applications of zkSNARKs, such as decentralized private currency and computation systems. In this work we design and implement privacy-preserving delegation protocols for zkSNARKs with universal setup. Our protocols enable a prover to outsource proof generation to a set of workers, so that if at least one worker does not col-lude with other workers, no private information is revealed to any worker. Our protocols achieve security against malicious workers without relying on heavyweight cryptographic tools. We implement and evaluate our delegation protocols for a state-of-the-art zkSNARK in a variety of computational and bandwidth settings, and demonstrate that our protocols are concretely efficient. When compared to local proving, using our protocols to delegate proof generation from a recent smartphone (a) reduces end-to-end latency by up to 26 × , (b) lowers the delegator’s active computation time by up to 1447 × , and (c) enables proving up to 256 × larger instances.
Investigation of Ionospheric Small‐Scale Plasma Structures Associated With Particle Precipitation

We investigate the role of auroral particle precipitation in small‐scale (below hundreds of meters) plasma structuring in the auroral ionosphere over the Arctic. In this scope, we analyze together data recorded by an Ionospheric Scintillation Monitor Receiver (ISMR) of Global Navigation Satellite System (GNSS) signals and by an All‐Sky Imager located in Longyearbyen, Svalbard (Norway). We leverage on the raw GNSS samples provided at 50 Hz by the ISMR to evaluate amplitude and phase scintillation indices at 1 s time resolution and the Ionosphere‐Free Linear Combination at 20 ms time resolution. The simultaneous use of the 1 s GNSS‐based scintillation indices allows identifying the scale size of the irregularities involved in plasma structuring in the range of small (up to few hundreds of meters) and medium‐scale size ranges (up to few kilometers) for GNSS frequencies and observational geometry. Additionally, they allow identifying the diffractive and refractive nature of fluctuations on the recorded GNSS signals. Six strong auroral events and their effects on plasma structuring are studied. Plasma structuring down to scales of hundreds of meters is seen when strong gradients in auroral emissions at 557.7 nm cross the line of sight between the GNSS satellite and receiver. Local magnetic field measurements confirm small‐scale structuring processes coinciding with intensification of ionospheric currents. Since 557.7 nm emissions primarily originate from the ionospheric E‐region, plasma instabilities from particle precipitation at E‐region altitudes are considered to be responsible for the signatures of small‐scale plasma structuring highlighted in the GNSS scintillation data.
43 Recent Developments of Surface Parameterization Methods Using Quasi-conformal Geometry

Surface parameterization is of fundamental importance for many tasks in computer vision and imaging. In recent years, computational quasi-conformal geometry has become an emerging tool for the design of efﬁcient and accurate parameterization methods for both surface meshes and point clouds. More speciﬁcally, using quasi-conformal (QC) theory, it is possible to reduce the geometric distortion and achieve conformal parameterizations for surfaces with different topology easily. It is also possible to achieve surface parameterizations that satisfy certain prescribed conditions, such as landmark constraints, with a minimal quasi-conformal distortion. In this article, we give an overview of the recent advances in surface parameterization using quasi-conformal geometry.
Super-resolution for face image with an improved K-NN search strategy

Recently, neighbor embedding based face super-resolution (SR) methods have shown the ability for achieving high-quality face images, those methods are based on the assumption that the same neighborhoods are preserved in both low-resolution (LR) training set and high-resolution (HR) training set. However, due to the “one-to-many” mapping between the LR image and HR ones in practice, the neighborhood relationship of the LR patch in LR space is quite different with that of the HR counterpart, that is to say the neighborhood relationship obtained is not true. In this paper, we explore a novel and effective re-identified K-nearest neighbor (RIKNN) method to search neighbors of LR patch. Compared with other methods, our method uses the geometrical information of LR manifold and HR manifold simultaneously. In particular, it searches K-NN of LR patch in the LR space and refines the searching results by re-identifying in the HR space, thus giving rise to accurate K-NN and improved performance. A statistical analysis of the influence of the training set size and nearest neighbor number is given, experimental results on some public face databases show the superiority of our proposed scheme over state-of-the-art face hallucination approaches in terms of subjective and objective results as well as computational complexity.
swissBERT : A Ready-to-Use Multitask Transformer

In this paper, we propose swissBERT , a multi-task learning approach that builds on BERT and incorporates several extensions, including direct encoding of sentence pairs, automatic weighted loss, cosine similarity loss, gradient surgery, biased task sampling, and weight sharing. We first demonstrate the effectiveness of minBERT in transfer learning for downstream tasks on sentiment analysis datasets through individual fine-tuning. We then extend minBERT to simultaneously perform three different downstream tasks using multitask fine-tuning, including sentiment analysis, paraphrase detection, and semantic textual similarity. Our experiments show that direct encoding of sentence pairs and auto weighted loss are highly effective extensions that consistently improve the model’s performance across tasks. Our experiments demonstrate an overall dev accuracy of 0.754 and an overall test accuracy of 0.758 on the leaderboard.