Miracle-dz/UniFAR-FTUs huggingface dataset:
Vision and Language Learning: From Image Captioning and Visual Question Answering towards Embodied Agents

Each time we ask for an object, describe a scene, follow directions or read a document containing images or figures, we are converting information between visual and linguistic representations. Indeed, for many tasks it is essential to reason jointly over visual and linguistic information. People do this with ease, typically without even noticing. Intelligent systems that perform useful tasks in unstructured situations, and interact with people, will also require this ability. In this thesis, we focus on the joint modelling of visual and linguistic information using deep neural networks. We begin by considering the challenging problem of automatically describing the content of an image in natural language, i.e., image captioning. Although there is considerable interest in this task, progress is hindered by the difficulty of evaluating the generated captions. Our first contribution is a new automatic image caption evaluation metric that measures the quality of generated captions by analysing their semantic content. Extensive evaluations across a range of models and datasets indicate that our metric, dubbed SPICE, shows high correlation with human judgements. Armed with a more effective evaluation metric, we address the challenge of image captioning. Visual attention mechanisms have been widely adopted in image captioning and visual question answering (VQA) architectures to facilitate fine-grained visual processing. We extend existing approaches by proposing a bottom-up and top-down attention mechanism that enables attention to be focused at the level of objects and other salient image regions, which is the natural basis for attention to be considered. Applying this approach to image captioning we achieve state of the art results on the COCO test server. Demonstrating the broad applicability of the method, applying the same approach to VQA we obtain first place in the 2017 VQA Challenge. Despite these advances, recurrent neural network (RNN) image captioning models typically do not generalise well to out-of-domain images containing novel scenes or objects. This limitation severely hinders the use of these models in real applications. To address this problem, we propose constrained beam search, an approximate search algorithm that enforces constraints over RNN output sequences. Using this approach, we show that existing RNN captioning architectures can take advantage of side information such as object detector outputs and ground-truth image anno-
Abductive innovations in information security policy development: an ethnographic study

ABSTRACT Developing organisational information security (InfoSec) policies that account for international best practices but are contextual is as much an opportunity for improving InfoSec as it is a challenge. Previous research indicates that organisations should create InfoSec policies based on best practices (top-down) and simultaneously encourages participatory development (bottom-up). These contradictory suggestions place managers in a dilemma: Should they follow a top-down or bottom-up approach? In this research, we build on an ethnographic approach to study how an innovative engineering company (MachineryCorp) managed the contradiction when the firm developed an InfoSec policy. Drawing on the dialectical theory of organisations as a lens, the findings suggest the InfoSec policy development is a recurrent process consisting of three phases: (1) drawing interpretations of InfoSec requirements from best practices (deductive adoption) and (2) constructing possibilities for local implementation (inductive adjustment) (3) that engender tensions between best practices and local contingencies facilitating innovative local resolutions (synthetic innovation). We call this process abductive innovation. At MachineryCorp, a triangle of tensions surfaced due to economic realities, infrastructure affordances, and social arrangements, and were necessary in explaining how the InfoSec policy gradually and iteratively materialised and resulted in an organisationally contingent policy.
Multi-Weighted Co-Occurrence Descriptor Encoding for Vein Recognition

Despite being highly secure, vein recognition suffers from the high inter-class similarity and intra-class variation resulting from the uncontrolled image capture, making the design of discriminative and robust representation very important. The recent success of convolutional neural network (CNN) for various image understanding tasks makes it a promising method for feature extraction. However, limited variability in small-scale datasets leads to systems derived from the direct training or fine-tuning not transferable and unreliable for practical biometric applications. This motivates the design of a multi-weighted co-occurrence descriptor encoding (MWCDE) model for vein recognition. Instead of directly conducting a feed-forward operation with a pre-trained CNN for obtaining the semantic features from the fully connected layers, co-occurrence features among convolutional filters are modeled first in MWCDE by a simple convolution between an indicator filter in a higher layer with a to-be-reweighted filter in a lower layer, and a redundancy-driven indicator filter selection algorithm is designed for filtering out some ambiguous representations. Second, another hard feature weighting strategy with a binary masking scheme is proposed for discarding noisy background and feature redundancy. The selected high-order descriptors are then embedded and aggregated into the compact feature vectors with a saliency driven spatial weighted Fisher vector algorithm, followed by the introduction of a generalized support vector machine for recognition. Extensive experiments with three benchmark vein datasets demonstrate that the proposed framework can achieve state-of-the-art results, and an additional experiment with the PolyU multispectral palmprint database illustrates its generalization ability. Code is available at $\langle $ https://github.com/RobinCSIRO/MWCDE-for-Vein-Recognition $\rangle $ .
Abstractive Text Summarization Using Hierarchical Reinforcement Learning

Author(s): Koupaee, Mahnaz | Advisor(s): Wang, William | Abstract: Sequence-to-sequence models have recently gained the state of the art performance in summarization. However, not too many large-scale high-quality datasets are available and almost all the available ones are mainly news articles with the specific writing style. Moreover, abstractive human-style systems involving a description of the content at a deeper level require data with higher levels of abstraction.On the other hand, attention-based sequence-to-sequence neural networks optimizing log-likelihoods at word-level or discrete metrics such as ROUGE at sequence-level has achieved promising results on abstractive text summarization but they are far from perfect: the first group of models may fail to handle out of vocabulary words and often produce repetitive words and incorrect facts. The latter methods using reinforcement training while beating the state of the art methods in terms of discrete evaluation metrics, produce non-readable, sometimes irrelevant summaries. We initially present WikiHow, a dataset of more than 230,000 article and summary pairs extracted and constructed from an online knowledge base written by different human authors. The articles span a wide range of topics and therefore represent high diversity styles. We also evaluate the performance of the existing methods on WikiHow to present its challenges and set some baselines to further improve it.Moreover, to overcome the problems of existing summarization systems, we propose a novel hierarchical reinforcement learning architecture which makes decisions in two steps: the high-level policy decides on the sub-goal for generating the next chunk of summary and the low-level policy performs primitive actions to fulfill the specified goal. By reinforcing summarization at different levels, our proposed model outperforms the existing approaches in terms of ROUGE and METEOR scores.
Genetic algorithm based sentence packaging in natural language text generation

Sentence packaging is an important task in natural language text generation which could be treated as a particular kind of a community detection problem. We propose an approach based on genetic algorithm and predictive machine learning models to advance it. The approach allows handling large ontological and semantic structures in a form of a graph to produce well-formed sentences. The results of experiments showed that the genetic algorithm optimizing the modularity measure gives comparable results to ones achieved by a traditional community detection algorithm and outperforms it on a collection of relatively short texts. The design of an approach allows for further introducing linguistic characteristics into a fitness function that gives it a high potential to increase the quality of detected packages while taking into account the specificity of the domain.
Collab4All: a method to foster inclusion in computer-supported collaborative work.

During the last decades, the research field of Computer-Supported Collaborative Work has made impressive progress in proposing theories, techniques, and tools to support collaborative work in different settings. Nevertheless, despite the advances in the field and its branches, individuals with impairments still face inclusion challenges when performing computer-supported collaborative activities. In order to take part in such type of activities, individuals are expected to have skills and knowledge that may be affected by impairments. In the beginning of this PhD research, we studied means to include an individual with blindness in diagram-based activities conducted as part of software engineering lectures taught by the thesis author. As part of that, we noticed that researchers and practitioners generally did not follow any collaboration-oriented method to address accessibility issues when designing or redesigning collaboration supportive systems, also known as groupware. In this context, we decided to focus our attention on this gap. As a result, we developed Collab4All, a method to support the identification of potential challenges as well as alternative solutions that may arise when designing or redesigning accessible and inclusive groupware. As a proof of concept, we applied Collab4All to the aforementioned scenario of including individuals with blindness in diagram-related learning activities. Based on our findings, we documented some suggestions to researchers and developers working in this application scenario. The present document contains detailed information about this research process. We consider that the method proposed in this thesis, its application results and the groupware we developed, named Model2gether, are relevant contributions to the field of computer-supported collaborative work involving people with impairments.
Variational Autoencoder Based Estimation Of Distribution Algorithms And Applications To Individual Based Ecosystem Modeling Using EcoSim

Individual based modeling provides a bottom up approach wherein interactions give rise to high-level phenomena in patterns equivalent to those found in nature. This method generates an immense amount of data through artificial simulation and can be made tractable by machine learning where multidimensional data is optimized and transformed. Using individual based modeling platform known as EcoSim, we modeled the abilities of elitist sexual selection and communication of fear. Data received from these experiments was reduced in dimension through use of a novel algorithm proposed by us: Variational Autoencoder based Estimation of Distribution Algorithms with Population Queue and Adaptive Variance Scaling (VAE-EDA-Q AVS). We constructed a novel Estimation of Distribution Algorithm (EDA) by extending generative models known as variational autoencoders (VAE). VAE-EDA-Q, proposed by us, smooths the data generation process using an iteratively updated queue (Q) of populations. Adaptive Variance Scaling (AVS) dynamically updates the variance at which models are sampled based on fitness. The combination of VAE-EDA-Q with AVS demonstrates high computational efficiency and requires few fitness evaluations. We extended VAE-EDA-Q AVS to act as a feature reducing wrapper method in conjunction with C4.5 Decision trees to reduce the dimensionality of data. The relationship between sexual selection, random selection, and speciation is a contested topic. Supporting evidence suggests sexual selection to drive speciation.
MāOri Loanwords: A Corpus of New Zealand English Tweets

Māori loanwords are widely used in New Zealand English for various social functions by New Zealanders within and outside of the Māori community. Motivated by the lack of linguistic resources for studying how Māori loanwords are used in social media, we present a new corpus of New Zealand English tweets. We collected tweets containing selected Māori words that are likely to be known by New Zealanders who do not speak Māori. Since over 30% of these words turned out to be irrelevant, we manually annotated a sample of our tweets into relevant and irrelevant categories. This data was used to train machine learning models to automatically filter out irrelevant tweets.
Bayesian Model Based Approaches In The Analysis Of Chromatin Structure And Motif Discovery

RITEN MITRA: Bayesian Model Based Approaches In The Analysis Of Chromatin Structure And Motif Discovery. (Under the direction of Dr P.K.Sen.) Efficient detection of transcription factor (TF) binding sites is an important and unsolved problem in computational genomics. Recently, due to the poor predictive ability of motif finding algorithms, along with the recent proliferation of high-throughput genomic technologies, there has been a drive to utilize secondary information, such as the positioning of nucleosomes, for improving predictions. Nucleosomes prevent transcription factor binding at those sites by blocking the TF access to the DNA. We aimed to construct an accurate map of nucleosome-free regions (NFRs), based on data from high-throughput genomic tiling arrays in yeast. Direct use of Hidden Markov Models are not always applicable due to variable-sized gaps and missing data. So we have extended the hidden Markov model procedure to a continuous time version while efficiently incorporating DNA sequence features that are relevant to nucleosome formation. Simulation studies and an application to a yeast nucleosomal assay demonstrate the advantages of the new method. The established biological role of nucleosomes in relation to TF binding, led us to formulate a joint model in the fourth chapter. The algorithm was implemented on the FAIRE data set, and comparisons were made with existing motif search algorithms. The fifth chapter deals with HMM asymptotics. We obtained results on consistency asymptotic normality and contiguity of a hidden Markov model. These have helped our inference on the convergence properties of the posterior and the consistency of the Bayesian posterior estimates. This has led to the conclusion that the Bayesian inference of a HMM run on sufficiently large datasets ( which is typical, in the case of genomic data) leads us very close to the underlying true parameters, as in the case of iid models. The result is fairly general in nature to provide the justification for HMM inference in a wide variety of datasets.
Sentiment Classification Using Document Embeddings Trained with Cosine Similarity

In document-level sentiment classification, each document must be mapped to a fixed length vector. Document embedding models map each document to a dense, low-dimensional vector in continuous vector space. This paper proposes training document embeddings using cosine similarity instead of dot product. Experiments on the IMDB dataset show that accuracy is improved when using cosine similarity compared to using dot product, while using feature combination with Naive Bayes weighted bag of n-grams achieves a new state of the art accuracy of 97.42%. Code to reproduce all experiments is available at https://github.com/tanthongtan/dv-cosine
Towards implicit learning of system-assigned authentication tokens

People tend to choose predictable passwords which are vulnerable to guessing attacks. To combat the security issue, system-assigned authentication keys were proposed, but this comes at a cost to memorability. In this thesis, I explore two different approaches to improve memorability of system-assigned keys through implicit learning: one that directly uses implicit memory alone, and another that indirectly uses implicit memory as a method to reinforce explicit memory. I first explore the feasibility of direct implicit learning-based authentication secrets, Tacit Secrets: system-assigned passwords that you can remember, but cannot write down or otherwise communicate. I design an approach to creating Tacit Secrets based on Contextual Cueing, an implicit learning method previously studied in the cognitive psychology literature. My feasibility study involving 30 participants indicates that my approach has strong security properties: resistance to brute-force attacks, online attacks, classical phishing attacks, and some coercion attacks. It also offers protection against leaks from other verifiers as the secrets are system-assigned. My approach also has a high login success rate and low false positive rates. I explore the trade-offs of different configurations of my design and provide insight into valuable directions for future work. In light of the promising results of Tacit Secrets, I propose a novel idea for training users system-assigned passphrases using implicit learning indirectly. Passphrases are passwords consisting of multiple words, initially introduced as more secure authentication keys that people could recall. Unfortunately, people’s propensity is to choose predictable natural language patterns in passphrases, again resulting in vulnerability to guessing attacks. Making
Automatic vertebrae localization, identification, and segmentation using deep learning and statistical models

Automatic localization and identification of vertebrae in medical images of the spine are core requirements for building computer-aided systems for spine diagnosis. Automated algorithms for segmentation of vertebral structures can also benefit these systems for diagnosis of a range of spine pathologies. The fundamental challenges associated with the above-stated tasks arise from the repetitive nature of vertebral structures, restrictions in field of view, presence of spine pathologies or surgical implants, and poor contrast of the target structures in some imaging modalities. This thesis presents an automatic method for localization, identification, and segmentation of vertebrae in volumetric computed tomography (CT) scans and magnetic resonance (MR) images of the spine. The method makes no assumptions about which section of the vertebral column is visible in the image. An efficient deep learning approach is used to predict the location of each vertebra based on its contextual information in the image. Then, a statistical multi-vertebrae model is initialized by the localized vertebrae from the previous step. An iterative expectation maximization technique is used to register the statistical multi-vertebrae model to the edge points of the image in order to achieve a fast and reliable segmentation of vertebral bodies. State-of-the-art results are obtained for vertebrae localization in a public dataset of 224 arbitrary-field-of-view CT scans of pathological cases. Promising results are also obtained from quantitative evaluation of the automated segmentation method on volumetric MR images of the spine.
Pricing American Options Under High-Dimensional Models with Recursive Adaptive Sparse Expectations

We introduce a novel numerical framework for pricing American options in high dimensions. Our scheme manages to alleviate the problem of dimension scaling through the use of adaptive sparse grids. We approximate the value function with a low number of points and recursively apply fast approximations of the expectation operator from an exercise period to the previous period. Given that available option databases gather several thousands of prices, there is a clear need for fast approaches in empirical work. Our method processes an entire cross-section of options in a single execution and offers an immediate solution to the estimation of hedging coefficients through finite differences. It thereby brings valuable advantages over Monte Carlo simulations, which are usually considered to be the tool of choice in high dimensions, and satisfies the need for fast computation in empirical work with current databases containing thousands of prices. We benchmark our algorithm under the canonical model of Black and Scholes and the stochastic volatility model of Heston, the latter in the presence of discrete dividends. We illustrate the massive improvement of complexity scaling over dense grids with a basket option study including up to eight underlying assets. We show how the high degree of parallelism of our scheme makes it suitable for deployment on massively parallel computing units to scale to higher dimensions or further speed up the solution process.
Text Categorization by Learning Predominant Sense of Words as Auxiliary Task

Distributions of the senses of words are often highly skewed and give a strong influence of the domain of a document. This paper follows the assumption and presents a method for text categorization by leveraging the predominant sense of words depending on the domain, i.e., domain-specific senses. The key idea is that the features learned from predominant senses are possible to discriminate the domain of the document and thus improve the overall performance of text categorization. We propose multi-task learning framework based on the neural network model, transformer, which trains a model to simultaneously categorize documents and predicts a predominant sense for each word. The experimental results using four benchmark datasets show that our method is comparable to the state-of-the-art categorization approach, especially our model works well for categorization of multi-label documents.