dataesr/scientific-paragraphs-categorization huggingface dataset:
Documents per year and sources Fig. 5  illustrates the distribution of documents from 2010 to October 2020 (as of the date of this work). The number of publications per year varies from 11 to 28. Regarding sources, 150 sources were found from the dataset, including conference proceedings and journals. The International Journal of Computer Integrated Manufacturing is the source with the most publications of 8 documents, while Automation in Construction is the most cited source (355), Computers in Industry in second place in terms of publications or citations. Table  3  exhibits the top 10 cited journals of the dataset, with the number of publications. Note that the sources are ranked by citations because of the approximate number of documents. The results could help readers efficiently identify the most important and influential journals related to the research problem.
Keyword occurrence analysis In order to perceive the research foci in the past decade, we analyze the most frequently used keywords via the co-occurrence analysis in VOSviewer. Table  4  exhibits the top 75 (with minimum 5 occurrences in the dataset) out of 2062 keywords in total. Removing first two terms "interoperability" and "supply chain" that are the keywords of the query, the top 10 technical terms are: semantics, ontology, information management, information systems, information services, service oriented architecture (SOA), web service, decision making, life cycle, IoT. Further, the keywords are grouped into five clusters as displayed in the table (by the best-performance  resolution setting in Vosviewer after running different configurations). It can be observed that: Cluster (1) of 16 keywords mainly concerning information sharing, communication; Cluster (2) of 16 keywords mainly related to design, life cycle, and sustainability; Cluster (3) of 15 keywords mainly concerning data format, handling and storage; Cluster (4) of 15 keywords mainly related to collaboration and competition in interoperable business and SC; Cluster (5) of 13 keywords mainly concerning the technologies of semantic web and virtual agents.
Other disruptive solutions include anonymous data exchange via blockchain, and privacy-preserving machine learning methods, that will be discussed later. Challenge 4: Product-oriented and order-oriented data. This challenge is concerned with the generation, collection and valuation of data in exchange process. In other words, it should be figured out what data is needed and why. Data can be collected from the objects (assets, products, etc. ), or from business process (orders, sales, etc. ). For the former, IoT technologies are massively used, and commonly applied in product design and life cycle such as intelligent product  (Barbosa et al. , 2016). For the latter, data-driven decision making techniques are often employed to take advantage of the (shared) data in logistics operations, for example, intelligent logistics involving customer data  (McFarlane et al.
g. , gradient) to the central algorithm for global optimization, according to  (Yang et al. , 2019). Furthermore, recent research proposes to combine federated learning and differential privacy methods to improve the performance  (Rodríguez-Barroso et al. , 2020). Generally, federated learning can be applied to different types of distributed and decentralized data sources, such as IoT devices, company databases, clouds. Thanks to the data privacy-preserving mechanism, its potential to federated logistics networks (like PI) should be tremendous. But until now no sufficient attention has been paid to this avenue. On the practical side, GS1 is a recognized organization for logistics data standards, such as RFID, EPCIS, and SSCC standards. Beyond those of GS1, there exists many other standards at operational level.
How to connect these standards, and what is the impact on logistics performance, are questions still needing to be investigated. Regarding data communications, using an API or a central data platform are the widespread conventional approaches. Some recent or undergoing projects can be cited here for a best-practices survey. For example, Clusters 2. 0 (clusters20. eu), DataPorts (dataports-project. eu), FENIX (fenixnetwork. eu), and LOGISTAR (logistar-project. eu), in Europe, Freight Share Lab (freightsharelab. com) in UK, and Cainiao Network in China. Furthermore, the graph paradigm is an approach increasingly used for managing and connecting databases, for example the graph database management systems like Amazon Neptune, Microsoft Graph, and Neo4j (Angles, 2018).
g. , life cycle assessment  (Lim et al. , 2020) , maintenance  (Errandonea et al. , 2020) , BIM in construction industry  (Greif et al. , 2020). Practical examples can be found in some projects such as DISpATch (dispatch-project. be) in Belgium, the EU project LEAD (leadproject. eu), or the IoT platform Thing'in (tech. thinginthefuture. com) by Orange in France. The information and communication technologies for DSCT are similar to those for autonomous systems as depicted above, but requiring more effort on virtual-physical communication and real-time data processing. To this end, new standardization technologies, such as semantics (viz. web application framework for machine-readability) and ontology (viz. open vocabularies to describe area of interest), are attracting particular attention in the DT related research streams  (Grubic and Fan, 2010).
Conclusion This paper investigated digital interoperability in logistics and supply chain management (LSCM), with particular reference to Physical Internet (PI) paradigm. We first conducted a bibliometric analysis to track the critical topics and trends, based on a dataset of 208 papers from the related literature. The statistical results reveal that the issue of digital interoperability in LSCM has been deeply studied in the past decade, and a number of solutions and concepts have been developed. However, there appears to be a lack of struc-tural classification to bridge the gap between the state-of-the-art solutions and PI because of its novelty. Therefore, we suggested four most important interoperability challenges induced by the PI paradigm, which served as a framework to classify the solutions.
, 2019)  allow KNN to be easily used on GPUs and integrated into neural networks. Further, the external memories are pre-encoded, so the information encoding is only computed once. As the external memories are kept fixed, they do not require any training to learn the memories along with the model. We can thus scale easily to larger memories by learning only the KNN-based read operation to identify relevant information from the memory. Our core contribution proposes an efficient, KNNbased Information Fetching (KIF) module that can access relevant external knowledge, combine knowledge from different sources, and integrate this information into standard sequence to sequence architectures. We apply these flexible modules to two dialog datasets that challenge generative models to leverage external information to write coherent, on-topic responses.
Both of our chosen tasks require models to leverage external information, such as information from Wikipedia or images, to engage in the conversation. We show that relevant information can be identified from hundreds of thousands of candidates in a multi-modal, multiknowledge-source setting to improve the performance of generative dialog models. Further, the output of the KIF modules is interpretable as specific humanreadable knowledge elements are selected, allowing users to better understand the information the generative model conditions upon when writing the subsequent utterance. On both datasets, we achieve state-ofthe-art results compared to generative models and find there is no statistically significant difference in the interestingness or human preference of our model output compared to state-of-the-art retrieval models.
To incorporate knowledge, models often concatenate a knowledge source E such as Wikipedia to x i and use attention modules to identify the most relevant knowledge. However, this approach is computationally intensive when handling large quantities of information. Further, attention mechanisms have been found to operate poorly over long sequences, as the mechanism becomes blurry due to the softmax and struggles to make fine-grained decisions  (Fan et al. , 2018b). The same is true for hierarchical approaches, which lack scalability. We augment Transformer sequence to sequence (seq2seq) networks on the encoder side with KIF to improve generative dialog models. We experiment on two dialog tasks, Wizard of Wikipedia  (Dinan et al. , 2018)  and Engaging ImageChat  (Shuster et al. , 2020). In both datasets, models must leverage information external to the dialog history alone -in Wizard of Wikipedia, the chat requires access to knowledgeable facts and in Engaging ImageChat, discussion about a specific image.
Wizard of Wikipedia The goal of the Wizard of Wikipedia dataset is to train knowledgeable agents that can chat in any domain. The dataset contains 1,365 various topics discussed in 18,430 dialogs in the training set, totalling 166,787 training utterances. Each topic is a general concept, such as dogs or ice cream, and is included as the first utterance of the conversation. The conversation is meant to be in-depth and detailed, so individual utterances must reference specific knowledge as a basis for the utterance. The knowledge takes the form of Wikipedia sentences. For example, the chat utterance I love Toy Story! It was released in 1995 would reference the Wikipedia sentence Toy Story is a 1995 American computer-animated buddy comedy [. ]. For each utterance, a set of sentences are identified by an information retrieval system, and the crowdworker selected one knowledge sentence as the basis for their utterance.
Knowledge Sources. Our model for Wizard of Wikipedia has access to two sources of external information, E 1 and E 2 : • E 1 is Wikipedia Knowledge provided by the dataset as evidence to support knowledgeable chitchat (initially curated by the information retrieval system used in  Dinan et al. (2018) ). The scale of this KNN search is to filter through an average of 34 sentences. The KIF module uses dialog features to fetch relevant knowledge to condition upon to generate the subsequent utterance. • E 2 is Training Utterances. To incorporate the benefits of retrieval-based dialog models to the generative setting, we use KIF to identify relevant utterances from the training set and take their responses as input. If many conversations about dogs have already occurred, models should be able to take advantage of these human-written examples to improve their generations.
Engaging ImageChat The goal of Engaging ImageChat is to create agents capable of chitchatting about images selected from the YFFC100M dataset  (Thomee et al. , 2016). The dataset contains 186,782 dialogs in the training set, each about a unique image, totalling 355,862 utterances. Agents are assigned one of 215 personalities (e. g. sweet, caring, excited) to increase engagingness. Previous work  (Shuster et al. , 2020 (Shuster et al. , , 2019) )  identified that both crowdworkers and models, when provided with personalities, produced more diverse, interesting responses, as evaluated by humans. We use a Multi-Modal neural network designed to handle both image input and text input. Following  Shuster et al. (2020) , the images are encoded using a pre-trained ResNeXt network  (Xie et al. , 2017).
To extract the final image representation, we project the 2048-dimensional output of the image encoder to 512dimensions using a deep multi-layer perceptron with ReLU activation units. The conversation history, which includes the one-word personality, is encoded with a Transformer encoder network. The image and conversation are integrated using the Multimodal-Sum-Combiner module proposed in  Shuster et al. (2020). Knowledge Sources. Our model for Engaging Im-ageChat has access to two sources of external information, E 1 and E 2 : Additional KNN Features. To identify relevant information from training utterances, we use the same dialog features as Wizard of Wikipedia in the KNN search step, with one modification: we add the personality provided by the dataset. We represent the personality feature as the personality word, such as caring, and embed it with the encoder M.
Implementation Details Parameter Settings. We use parl. ai (Miller et al. , 2017)  to implement our models. The data for both datasets used is available for download from parl. ai as well. We use byte-pair encoding  (Sennrich et al. , 2016)  to represent the text to better handle the rare word problem  (Dinan et al. , 2018; Fan et al. , 2018a). Our generative Transformer models have 8 encoder layers and 8 decoder layers, with FFN size 2048, embedding dimension 512, and 4 attention heads. We optimize using Adam (Kingma and Ba) and the inverse square root learning schedule  (Vaswani et al. , 2017)  with 10k warmup updates. The initial learning rate is 0. 0001 and we optimize for model perplexity. We use a dropout of 0. 5 and set gradient clipping to 0. 1. We set k = 5 for all cases. For both datasets, we model a vocabulary size of 54944 based on the BPE-based vocabulary from the Reddit pre-training.
We tuned the learning rate and batchsize hyperparameters together. Pre-training. We pre-train the Transformer seq2seq model used for both datasets on 250M comments from Reddit. The Reddit dataset was made available by pushshift. io. The comments are parsed to maintain conversational threads of users responding to each other, so the encoder network has been exposed to conversational context at training time. Note that the Reddit dataset does not include aspects such as personality, as those are unique to specific datasets such as Engaging ImageChat. The context size in pre-training is set to 512 tokens. The ResNeXt encoder used to model images for the Engaging ImageChat dataset was pretrained on 3. 5 billion images  (Mahajan et al. , 2018).
Evaluation Generation. We generate with beam search, setting the beam size to 4. We use 3-gram blocking. This technique disallows repeated n-grams from being generated multiple times and reduces repetition. Automatic Metrics. Following Dinan et al. (  2018 ), we compute F1, a metric of unigram overlap, between the generated utterance and the human-written reference utterance from the dataset. For generative models, utterances are generated using beam search. For retrieval models, the next utterance is predicted by ranking the entire set of training utterances, and the highest scoring utterance is chosen. In Wizard of Wikipedia, there are two test sets: a set of seen topics, or topics that have been seen at training time with new test-time dialogs. The second set is unseen, or topics that have not been encountered at all during training time.
Generative Baselines Generative Transformer MemNet  (Dinan et al. , 2018)  7. 1 + Reddit Pre-Training 12. 8 Retrieve and Refine  (Weston et al. , 2018)  13. 6 Response Generation with MR  (Qin et al. , 2019)  13. 2 KIF-Augmented Transformer 14. 4   Human Evaluation. We follow the setup and use the analysis questions proposed in the Acute-Eval dialog evaluation system  (Li et al. , 2019). For reproducibility, we adopt this existing evaluation setting that has been applied to several dialog datasets. We use the question wording suggested by Acute-Eval and follow their self-chat procedure and interface. As one of the original datasets assessed in this system was Wizard of Wikipedia, their evaluation setting extends naturally to ours. We collect 100 human-bot conversational dialogs on a crowdsourcing platform for both datasets.
Baselines We compare Transformers augmented with KIF to other existing approaches on Wizard of Wikipedia and Engaging ImageChat. The best approaches, judged by human evaluation, are retrieval models, the Retrieval Transformer Memory Network from  (Dinan et al. , 2018)  and the Retrieval Transformer from  Shuster et al. (2020). These have been shown to be strong baselines compared to other retrieval techniques based on TF-IDF  (Chen et al. , 2017). Thus, we report the existing retrieval models for both datasets, but focus on Human: I love watching disney movies! Do you like Disney? Model: I love disney, I love watching disney movies and different animations and characters that make them unique Fetched Knowledge: disney consumer products and interactive media -the company specializes in producing animated films Fetched Training Utterance: i love music, I love going to music shows and dancing and having fun Human: What's your favorite disney movie? There are so many good ones.
Model: Ohhh. I think toy story is a classic? It was first released in 1999. Fetched Knowledge: toy story 2 is a 1999 american computer-animated comedy film produced by pixar animation studios for walt disney pictures Fetched Training Utterance: Ohhh. What do people with color blindness do to cope with the effects? comparing to other generative baselines. We compare to three additional generative baselines. Note that in Wizard of Wikipedia, the construction of the dataset is that sentences of Wikipedia knowledge are provided with the utterances in a concatenated form. Models must identify the relevant information in this provided knowledge, or can access more Wikipedia knowledge beyond the provided sentences. The fol-lowing baseline methods always have access to the information provided in the dataset already, but no additional Wikipedia knowledge beyond that.
We implement a hybrid model  (Weston et al. , 2018)  that incorporates top retrieval candidates as additional input to Generative Transformer MemNets. In Retrieve and Refine, a fixed number of candidates are retrieved and concatenated to the conversational history in the encoder, making the input much longer. For both datasets, the Retrieve and Refine mechanism that fetches a fixed number of training utterances is added to the Generative Transformer MemNet with Reddit Pre-Training baseline. Unlike the KIF-Augmented Transformer, the retrieval is conducted with a separate model so there is no backpropagation to affect the retrieval. With KIF, models can alter the retrieved candidates by learning the mapping operator. Further, a fixed amount of information is always retrieved, without the capability to easily rescale to focus on specific candidates.
KIF modules have weighting mechanisms to focus more on certain information, and the modules are combined with gating so models can learn which knowledge sources are more important and adjust flexibly. Lastly, Retrieve and Refine is only used to retrieve one source of information: training set utterances. • Response Generation with MR. We implement the model proposed in  Qin et al. (2019) , which encodes the conversation history and document contextually with a biLSTM before generating the next dialog utterance. The initial model was applied to a machine reading task where a knowledge document was provided along with the conversation history. For Wizard of Wikipedia, we replace the knowledge document with the Wikipedia sentences provided in the dataset. The model then uses the conversation to identify the most relevant information in the document using a cross-attention mechanism.
For the Engaging Im-ageChat dataset, as there is no document provided with the dataset, we replace the expected document with the conversation history, and use the most recent utterance in the conversation to attend to the conversation history. We make an additional improvement to this baseline: in  Qin et al. (2019) , the embeddings used pre-trained CoVE vectors  (McCann et al. , 2017). We found our Reddit pre-trained Transformer embeddings to work more effectively as they are trained for dialog. Thus, we replace CoVE embeddings with domain-specific ones. All of Transformer generative baselines are initialized with the same pre-training on Reddit that we use for our models for fair comparison on modeling quality.
Results We describe the results of incorporating KIF modules into Transformer networks. We display an example conversation between a human and our model in We compare to various baselines using automatic and human evaluation, and discuss our experiments. We present various ablation settings to understand the key features that make our method function. 6. 1 KIF is Effective for Incorporating Knowledge Automatic Evaluation. Comparing KIF augmented Transformer networks to published baselines and Retrieve and Refine, we find improved results. For Wizard of Wikipedia, the improvement in F1 score over the best baseline is around 8 points (see Table 1). A major contributing factor is the construction of the dataset -as each dialog turn is grounded in a specific knowledge sentence from Wikipedia, improving the ability to identify the relevant fact strongly improves performance.
Contrasting the results from the seen and unseen test sets in Table  1 , the improvement on unseen is worse -it is harder to fetch training utterances for unseen topics. While Imagechat has no explicit dependency on knowledge, we still see a 2 point improvement compared to the Generative Transformer MemNet (with the additional Reddit pre-training), indicating that KIF can be generally useful (see Table  2 ). Compared to an even stronger baseline that we tune in this work, Retrieve and Refine, we see 1 point improvement. Human Evaluation. Results are shown in Figure  2. On both datasets, we find there is large improvement over existing generative models (green bars) that is statistically significant for some of the evaluation questions. Evaluators agree that KIF-augmented Transformers are generally more coherent and humansounding compared to the Generative MemNet.
Compared to generative models, which we focus on improving, retrieval models often produce longer text with more interesting, nuanced vocabulary usage, and do not make generation mistakes such as repetition. These factors often lead to the stronger performance of retrieval models. A surprising result is that KIF-augmented Transformers are more human sounding than retrieval models on Wizard of Wikipedia. This is because the dataset's utterances are long and factual due to the tendency of crowdworkers to copy Wikipedia. Sometimes humans chatting with the retrieval bot would respond uh. that's an interesting fact? Otherwise, our model scores similarly to retrieval models, with most evaluations not having statistically significant difference. We conduct a second evaluation on the Unseen Test Set of the Wizard of Wikipedia dataset.
For example, the model copies the training utterance Ohhh. what do people with color blindness do to cope with the effects? and starts the model generation with Ohhh. and continues with the question i think toy story is a classic? following the form of the selected training utterance. Figure  5  displays the top-3 fetched training set utterances and knowledge sentences on the Wizard of Wikipedia dataset when responding to a human utterance. KIF modules can identify multiple relevant items. In response to the human question about blue skies the 1946 movie the model identifies both the comedy film and the band. Finally, the elements retrieved by KIF modules provide a more interpretable understanding of what the model is conditioning upon to generate a dialog response. In Table  3   while changing the fetched training utterance changes the form of the generated sentence.