DOI: 10.5061/dryad.cvdncjtb5
title: Data from: Cornerstones are the key stones: Using interpretable machine learning to probe the clogging process in 2D granular hoppers

description: This dataset contains the information recorded for approximately 50,000
 hopper flows under varying conditions. Each file represents a single flow
 event, from beginning of flow to final clog forming. The files are matlab
 structures, containing the positions, radii, frame number, tracked
 particle ID's and velocities for every grain in the camera field of
 view throughout the flow. Other values, such as the position of the outlet
 and the total mass ejected during the flow are also included.
description: Data are the tracked positions of grains throughout individual
 hopper flows (beginning of flow to final clog). A camera records images of
 grains near the outlet at 130 frames per second, which was then analyzed
 using matlab code. The grain centers were located and species (size)
 identified; this information was then fed into a tracking algorithm to
 uniquely identify grains throughout flow. From these unique identifiers,
 velocity and ejected mass information was calculated. 
description: # Data from: Cornerstones are the key stones: Using interpretable machine
 learning to probe the clogging process in 2D granular hoppers ## Grain
 positions during hopper flow for Autohopper Data are the grain positions,
 velocities, sizes, and IDs throughout the flow from the breaking of a clog
 to a new stable clog forming in a granular hopper. Grains are a
 tridisperse mixture of discs made from anti-static Ultra High Molecular
 Weight Polyethylene. The hopper is two vertical sheets of plexiglass with
 enough space between them to admit a single layer of discs with minimal
 displacement. Images of the grains during flow were recorded using a
 camera, and grain locations were tracked using custom MATLAB code. ##
 Description of the data and file structure The data are separated into
 three folders reflecting different experimental conditions. The folder
 FixedParticle.zip contains 5551 flows in which a fixed grain was inserted
 near the outlet, systematically affecting the flow.
 SingleOutletNoFixedParticle.zip contains 35116 flows at a single outlet
 size without this fixed grain. This represents the dataset primarily used
 for our machine-learning analysis. OtherOutletsNoFixedParticle.zip
 contains approximately 7263 flows at a variety of other outlet sizes,
 again without the fixed grain. Data are MATLAB structures with a variety
 of packaged information. All spatial units (positions, radii) are in
 pixels. The frame number is in frames. Time stamps are in seconds. Basic
 tracking information is stored in 'tinfo_header', and describes
 the organization of the data stored in 'trackinfo'. This data is
 organized into columns listed in 'tinfo_header', repeated here:
 1. x position 2. y position 3. time stamp 4. frame number 5. grain species
 (size category) 6. x displacement from the previous frame 7. y
 displacement from the previous frame 8. grain ID Additional parameters
 relevant to the system are stored in 'parameters'. These are the
 radii of the grain species in pixels ('rad_small',
 'rad_medium', 'rad_large', 'rads'), and
 recording parameters ('exposure', 'fps',
 'display_rate'). Final information is stored in
 'massflow'. The variables are: 1. M: the cumulative ejected area
 per frame 2. T: the timestamp of each frame 3. D: the size of the outlet
 in pixels 4. cpt: the coordinates of the center of the outlet 5. R: the
 radius of a large grain, in pixels 6. flow_issue: if something happened in
 the lab such that this trial should be discarded One final flag is stored
 in 'fixedParticle'. If this value is set to 1, there was a fixed
 particle inserted near the bottom edge of the hopper. If set to 0, all
 grains in the system are free to flow. The naming convention of data
 folders should preclude the mixing of these states. Other stored data
 structures were used during original data processing and can be ignored.
 In addition to the raw data, processed versions of the fixed grain data
 are also provided in the BinCenters.txt, Dimensionless_M.txt, and
 Dimensionless_M_std_error.txt files. This effectively recreates the
 histogram of data in Figure 4. BinCenters.txt are the center of each
 histogram bin for the fixed grain position. Dimensionless_M.txt
 and Dimensionless_M_std_error.txt are the ejected mass per fixed grain
 position bin and its standard error, respectively. These values are
 normalized by the average ejected mass of the system at the same outlet
 size without a fixed grain. ## Code/Software We separately provide two
 Python scripts and a Jupyter notebook that can be used to reorganize the
 data. The first, FlowClass_WithFixedGrain.py, is a class to take the
 MATLAB-formatted data and input it into Python. It also has helper
 functions used in the other two scripts, ClassSelectionScript.py and
 FixedParticleFigure.ipynb. The former, ClassSelectionScript.py, requires
 the packages numpy, pyvoro, scipy, and networkx, whereas
 FixedParticleFigure.ipynb requires numpy, matplotlib, os, pickle, and
 scipy. ClassSelectionScript.py takes a data path to a folder filled with
 AutoHopperData and creates a folder for each class of flow state we use in
 our paper (to be linked when the preprint is uploaded). For each flow in
 the source folder, a frame of each class is found if possible, and that
 state is saved as a .npy file to the respective folder. The structure of
 the data is each row representing a grain, with the columns containing the
 following information: 1. x position 2. y position 3. x displacement from
 the previous frame 4. y displacement from the previous frame 5. radius To
 run this script, it must be in the same folder as FlowClass_General.py,
 and it will create the class folders locally. The variable at the
 beginning of the script, filepath, should be a text string of the path to
 the folder containing the MATLAB files to be identified. In addition to
 the Python packages required for FlowClass_General.py,
 ClassSelectionScript.py requires os, shutil, and time.
 FixedParticleFigure.ipynb is an interactive Jupyter notebook that takes
 the analyzed fixed-grain data and recreates the analysis of Figure 4 in
 the paper. This relies on the already processed fixed grain positions and
 ejected mass data, stored in BinCenters.txt, Dimensionless_M.txt,
 and Dimensionless_M_std_error.txt. The Jupyter notebook should be in the
 same folder as FlowClass_General.py, and the filepaths to each of the
 above text files should be put in at the top of the notebook.