dataesr/scientific-paragraphs-categorization huggingface dataset:
Data Life Cycle Metamodel Informally, the life cycle model of a data item is the set of all the states it can have at any time, and the set of transitions that dictate what changes of state are possible. In this section we first give our notations and formal definitions for data life cycle models and present several developments: i) decoration of tokens to allow unique identification of data replicas, ii) data life cycle termination rules, that allow to detect errors and illegitimate operations and iii) how Active Data exposes a unified view of data life cycles even though they involve several heterogeneous systems.
Data Identification and Replication Packed with elements from Petri Networks, our life cycle model is now able to represent data states, transitions, and replication. However, we need a way to discriminate tokens and assign them identifiers that links to the real-life data. Systems discriminate data items they manage by assigning a unique identifier to each of them. Later, when the system needs to apply some treatment to a specific data item, it use its identifier to reference the targeted item. Data management systems use a variety of conventions for data identifiers; for example iRODS  [13]  assigns a property named R_DATA_ID of type Long to each file; HDFS  [14]  refers to files using URIs; database records are retrieved with a reference to the database, a reference to the table and the identifier of targeted tuple (typically an auto-incremented integer).
Constructing a new life cycle model The first step for using Active Data is to construct an object-oriented representation of the life cycle of data managed by a system. The root of the model is a class called LifeCycleModel; objects of this class link to Place and Transition objects. Constructing a LifeCycleModel requires to decide of a unique system identifier (SID) for the system. This class has methods for adding places and transitions, that also require names. LifeCycleModel myModel = new LifeCycleModel (" My System "); Place p1 = myModel. addPlace (" not executed "); Place p2 = myModel. addPlace (" executed "); Transition t = myModel. addTransition (" execute "); myModel. addArc (p1 , t); myModel. addArc (t , p2 );.
Publishing a new life cycle Active Data users manipulate LifeCycle objects. A LifeCycle represents a data item in the system -a file in our example; it links to the LifeCycleModel and contains all the tokens on the right places. All applications participating in the life cycle of a data item must create LifeCycle objects to link the files they manipulate to Active Data. After that, the service saves the state of the life cycles and is able to receive transition publications regarding them. Informing the service that a new data item has been created is called publishing a new life cycle. The arguments required to publish a new life cycle are the life cycle model and the data item unique identifier. ActiveDataClient client = ActiveDataClient. getInstance () ; LifeCycle lc = client. createAndPublishLifeCycle ( myModel , " xyz "); Sometimes users want to be informed that a new data item has been created in a particular system.
Subscribing to transitions from the model will delay the information because an arbitrarily long time can pass between the creation of a data item and its first transition. For this reason, an additional transition called create transition is automatically added to each life cycle model. When a client publishes a new life cycle, the service silently publishes its create transition. Clients can subscribe to this transition like to any other. Clients get the create transition from a life cycle model with a method call on a LifeCycleModel object: Transition create = appModel. getCreateTransition () ;.
Publishing transitions Newly created life cycles have only one token on their Created place. Applications must move to other places to reflect the progress of data items through their life cycle. All applications must notify Active Data of the operations they perform on data. This is called publishing a transition; transitions must be published as soon as possible after an operation is performed. Publishing a transition is a single call: LifeCycle lc = client. createAndPublishLifeCycle ( myModel , " xyz ");. Transition t = myModel. getTransition (" My system. t"); client. publishTransition (t , lc ); The first line is the same as before and just creates a life cycle; the last two lines get the Transition object by its name and publish it by calling the publishTransition(Transition, LifeCycle) method of the Active Data client interface; the transition to publish and the life cycle that is being updated are given as arguments.
The dealer -which gets a chance to examine every place and token and decide which to pick-is called internally by Active Data on the client side when a process is attempting to publish the transition it is attached to. If no dealer has been explicitly attached to the transition (which is the general case) a default transition dealer is present. The behavior of the default transition dealer is not defined for complex cases such as the one we are discussing; it is only guaranteed to consume the right number of tokens from the input places, and to produce the right number of tokens on the output places. If the number of output tokens is greater than the number of input tokens, the default dealer produces the necessary number of new tokens by calling produceNewToken(Token, Place). Note that a transition dealer may be attached to multiple transitions, but that a transition can only have one dealer attached.
Publishing a composition transition To inform Active Data of data passing from one system to another, a process (usually the one that initiates the operation) must publish a composition transition every time a data item is sent to an other system. Publishing a composition transition is similar to publishing a regular transition, except it requires an additional argument: the unique identifier of the data item in the new system. Listing 2 demonstrates how to publish the composition transition on figure  3. Here, the composition transition consumes the first token present on the p 2 place. This token's unique identifier is "1234"; the publishing process initiates the transfer of the data item from A to B. B returns the identifier it has locally assigned to the data item. When the publishing process publishes the composition transition with this identifier, Active Data creates a new token on the Created place of system B.
Transition handlers Transition handlers are the code provided by the programmer, that are to be executed whenever DLC transitions happen. A first transition handler was presented in Listing 1. A handler is an object that implements the TransitionHandler interface and the handler method. The method receives four arguments that provide the context necessary to write flexible code: transition is the object representing the transition that was published and caused this handler to be executed; isLocal is true only if the client running the handler is the same client that published the transition (this is valid because in Active Data every client can be both subscriber and publisher); inTokens the set of tokens that were consumed by the transition; outTokens the set of tokens that were produced by the transition.
The handler is able to examines the tokens consumed and produced by the transition, to extract their unique identifier and uses them to access the real data. Additionally, tokens also link to the complete LifeCycle object which allows to observe where the other tokens are located, and examine the complete state of the data. The Transition argument allows to tie the same handler code to several transitions. In addition, the same TransitionHandler object is used by Active Data every time in needs to be executed; in other words, transition handlers can be stateful. For example, a user may not want to be notified every time a transition is published; instead they might want the handler to notify them each n times the handler is executed instead.
Transition subscription The transition handlers must be attached to a transition to be executed after the transition is published. This action is called subscribing to a transition. Because many data items share the same life cycle model, the number of transitions being published can grow rapidly. However, users usually want their handlers to be executed only when it is relevant to them, and ignore everything else. Active Data allows programmers to cut down the number of transitions their code will react to by providing two kinds subscriptions: i) subscribing to a specific transition for any data item and ii) subscribing to a specific data item for any life cycle transitions. Paying close attention to the example handlers in Listings 1, we notice that the value of the Transition argument is never tested; in these cases it is not needed, because these handlers are meant to be subscribed to a single transition only.
There is a second type of subscription that allows to be notified of every transition, but only for a particular life cycle: The LifeCycle class allows to examine all places, transitions and tokens for any data item. Active Data offers a querying interface that allows any programmer to obtain the complete life cycle of any data item, no matter where their code is running. Users query a life cycle with a couple (SID, UID). Through composition, a life cycle is identified by as many such couples as there are systems in the life cycle. However, only one is necessary, allowing code with a very partial view expand it to the whole life cycle. For example, consider a storage system composed with an execution system. A client of the storage system would like to discover the state of the computation in which a particular file is involved.
The client knows that the identifier of the storage system is "storage" and that locally, the identifier of their file is, for example, the integer 36253. Listing 3 shows how the client can use Active Data to discover i) the identifier of their file in the execution system, and ii) the state of the application. for ( Token t: lifeCycle. getTokens (p)) print (" Token with id " + t. getUid () + " is on place " + p. getName () ; Listing 3: Querying the complete life cycle of a data item from a partial knowledge. Active Data returns the complete LifeCycle object with all the tokens in every system. The query feature is an important building block for data integration: Active Data integrates under a single namespace the identifiers of the same data from different systems that were not designed to collaborate and provides them to users and any process.
System Architecture The architecture of Active Data is illustrated by Figure  7 ; it is composed of the centralized Active Data Service and of clients communicating with the service using the client API. For the Active Data Service, managing life cycles has two functions: i) maintaining the current state of each life cycle and ii) allowing clients to query life cycles. The fact that the service is centralized allows it to easily maintain the consistency of life cycles. Clients of the Active Data Service are of kinds: • Data management systems and user applications are client of the Active Data Service when they publish transitions; sometimes the same client that publishes transitions can subscribe to handlers; • end users are clients of the Active Data Service when they query the state of life cycles and subscribe handlers to get notifications or run local code.
Hadoop  [18]  is a project maintained by the Apache Software Foundation that includes a widely used implementation of the MapReduce  [4]  programming model and a distributed file system called HDFS  [14]. When integrating Active Data into a legacy system, it is necessary to understand how the system exposes its life cycle, in order to detect and convey life cycle transitions. The main approach consists in relying on notifications provided by the system, and publishing these notifications in terms of Active Data transitions. For instance, inotify provides a notification service, which wakes up programs when files are altered on the file system. inotify informs about the new state of the file, thus it is easy to deduct the corresponding place, and publish the corresponding transition. When such a notification service is not available, a second approach is to look at the internals of the system.
For instance, iRODS relies on a PostgreSQL database. We implemented a trigger executed by the database each time a file is created or modified in iRODS. The trigger acts as an Active Data proxy and forwards only relevant life cycle transitions to the Active Data Service. A third approach is to deduce data-related events from a system's logs. In the case of Hadoop, for example, the submission, start and completion of jobs and tasks is reported on the job tracker and task trackers logs. A lightweight scraper is executed on every machine of the cluster, watches the logs and publishes the corresponding transitions. Overall, we think there exist many sources of information to obtain complete or partial representations of data life cycles in a non-intrusive fashion: logs, email notifications, databases and so forth.
For each system, we have to represent its data life cycle model. Systems that expose only partially their life cycle make this task intricate: this is the case for Globus Online and iRODS. As Globus Online source code is not available, we have an incomplete knowledge of the file transfer life cycle. However, Globus Online emails the user upon successful file transfer completion or failure. From this partial information, we reconstruct the life cycle model presented in Figure  8d  and enable Active Data users to monitor Globus Online transfers. With iRODS, we do not need the whole iRODS data life cycle and chose to represent only the portion that is relevant to our application (see  5. 3. 4). This model is represented in Figure  8c. Conversely, two data life cycles are complete: inotify, BitDew.
Hadoop benchmarks To complete the performance evaluation presented in the last subsection, we evaluate Active Data's performances against a real life workload. To this end we use the life cycle model of files in HDFS and Hadoop and run an intensive sorting benchmark on a 1TB dataset. Active Data must be able to handle the many transition publications induced by the map and reduce tasks. We execute the Terasort benchmark on a cluster of 45 nodes, each having two 4-core Intel Xeon L5520 CPUs running at 2. 26GHz, 32GB of memory and two 300GB hard drives. The nodes are interconnected with gigabyte ethernet. We run the Active Data Service alone on one node and we setup Hadoop to run a mapper or reducer on every core of the remaining nodes. Thus on this setup we run 280 mappers and we decide to run 70 reducers.
The Terasort benchmark runs in two phases: a random dataset is generated by a first job called Teragen that we do not monitor, then the actual Terasort job is executed to sort it. The Active Data Service records how many transition publications it performs every second, like in the last subsection. Figure  10  plots the resulting data; after a peak during the first few seconds of the job, the number of transitions per second is relatively low as map and reduce tasks progress. The first peak is due to the many Submit map, Submit reduce, Assign map and Assign reduce transitions that are published in a burst when the job tracker starts the job and partitions the workload. During this first peak, a maximum of 196 transitions per second is recorded. After this, transitions are published at a much slower pace of 3 per second on average with a standard deviation of only 5, as map and reduce tasks complete.
• Active Data can expose to the programmer a single data life cycle, even if data items are managed by several heterogeneous systems. The last scenario presents the construction of a unified life cycle model based on the composition of two different systems: iRods and Globus Online. Thanks to this model, we present an application which automatically keeps track of data items provenance when they move from one system to the other. We evaluate the uses-cases against four criteria that are representative of the systems and application class we support: • The unique high-level view of data life cycles offered by Active Data must allow users to implement cross-system optimizations; • The global namespace maintained by Active Data must allow to integrate in the same scope data objects from several non-cooperative systems; • The programming model must allow to program distributed data life cycle management tasks easily, benefiting from implicit parallelism; • The event driven model must allow to program applications able to react to changing data.
Collaborative Sensor Network This case study illustrates: i) the adaptability to legacy data management systems, ii) the ability to develop distributed applications that support independent data life cycles distributed over several local systems and iii) how easy it is to implement coordination between distributed nodes with Active Data. It is a common practice for applications acquiring data -for example from a sensor network-to apply some pre-processing before being pushed on a computing platform and archived. Pre-processing can be used to filter, compress data or remove invalid data. Such a sequence of operations can easily be scripted using ad-hoc languages or programs. Data throttling is also a common practice to reduce the amount of data injected in the system at a given time. Decentralize data throttling enables to reduce the load on the system by dropping data before they are injected.
As soon as a camera writes an image file on a node's filesystem, it is considered as a newly created data item and its life cycle begins. To capture these transitions on files, we use the inotify Linux kernel subsystem. Inotify allows to watch a directory and receive events about the files it contains. Events regard file creations, modifications, writes, movements and deletions. As inotify events represent filesystem events, and filesystems contain data (files) that are subject to transitions, we can represent inotify events with an Active Data model. Figure  8b  presents the inotify Active Data model, constructed using the method described in section 4. 3. The combination of Active Data and inotify creates a distributed inotify: all nodes can now coordinate based on transitions happening on other nodes' filesystems.
More precisely, nodes listen to two transitions triggered by the storage system, thanks to Active Data: • t 3 is observed by the master node. When this transition is triggered, the master node checks whether the transfer is local and whether it modifies an input chunk. Such case happens when the master puts all the data chunks in the storage system before launching the job. If both conditions are true, the transition handler flags the chunk as dirty. • t 1 is observed by the mappers. When this transition is fired, mappers check whether the transfer is distant and if one of their input chunks is modified. In this case, the transition handler on the mapper flags the chunk as dirty. To evaluate the performance of incremental MapReduce, we compare the time to process the full data set compared with the time to update the result after modifying a part of the dataset.
The experiment is configured as follows; the benchmark is the word count application running with 10 mappers and 5 reducers, the data set is 3. 2 GB split in 200 chunks. Table  4  presents the time to update the result with respect to the original computation time when a varying fraction of the dataset is modified. As expected, the less the dataset is modified, the less time it takes to update the result: it takes 27% of the original computation time to update the result when 20% of the data chunks are modified. However, there is an overhead due to the fact that the shuffle and the reduce phase are fully executed in our implementation. In addition, the modified chunks are not evenly distributed amongst the nodes, which provokes a load imbalance. Further optimizations would possibly decrease the overhead but would require significant modification of the MapReduce runtime.
A second transition handler is attached to iRODS's Put transition: it is executed when any iRODS data is written the first time, right after its creation. This handler requests the life cycle from the Active Data Service to see if it contains a Globus Online identifier. In such case, it uses the token identifier to query Globus Online and get file transfer information. To demonstrate our solution, file transfers are launched from a remote Globus Online endpoint to a local temporary storage every few seconds. We observe that when a transfer ends, it appears immediately in the iRODS data catalog with the correct Globus Online Task Id and meta-data information (endpoint, completion date, request time). Listing 4 shows the metadata set for one of these iRODS files after the transfer is done. The global and unique namespace provided by Active Data of datasets over heterogeneous and non-cooperative systems significantly simplifies the challenge of global provenance reconstruction.
Related Work In this section, we review existing programming models for data-centric and distributed applications as well as popular distributed storage systems and argue that they are inadequate for supporting DLCM applications. The Big Data challenge has renewed the approach to parallel programming in many aspects. MapReduce  [4] , introduced by Google in 2004, is a good example of this quest for new paradigms to simplify the processing and generation of large data sets. In the wake of MapReduce, several other data-centric languages have emerged, either as alternative, e. g, Dryad  [5]  for dataflow parallel computing, Allpairs  [22]  to perform pair-wise comparisons in large data sets, or as evolution of the paradigm, e. g. , PigLatin  [23]  to provide high level query interface on top of MapReduce, Twister  [24] , a framework for iterative MapReduce computations, to cite a few of them.
Although these works outline the necessity for investigating formal representations of data-sets, our context differs and objectives diverge as well. We want to capture the state of a dataset when it is deployed on the entire infrastructure as well as all legitimate possible future operations. There exist countless DLCM systems where Active Data could be implemented aside, but systems that provide high level interfaces to data management would be the ones that would benefit the most. We list some of them and give hints about possible improvement. Data attributes is a feature proposed by BitDew which allows the user to specify data behaviour such as fault-tolerance, replication, file transfer protocol or affinity placement. The implementation of Active Data and BitDew  [15]  together that we have proposed in this paper would allow the programmer to take advantage of the BitDew data attribute to wisely control and steer the distribution of data, while using Active Data to program applications that could react on each event happening during the distribution of the data set.
Experimental results We now describe some experimental results obtained from the K-Rb interferometer. All of the data presented in this work were recorded in a laboratory environment, with the interferometer beams aligned along the vertical direction, and with no anti-vibration platform. To compensate for the Doppler shift due to gravity, the frequency difference between Raman beams for interferometers j = 1 ≡ K and j = 2 ≡ Rb is chirped at a rate of α j k eff j g to account for the gravity-induced Doppler shift of the falling atoms. This modifies the total phase shift of the interferometers from eqs. (2), Φ j = S j a + φ j , to the following The last expression represents the case when both interferometers experience the same acceleration, a j = a = g, and the scale factors can be approximated as S j k eff j T 2 j.