Package: cox
Cox: An experimental design and analysis frameworkYou can find API Documentation on Coxhere, along with a copy of the
Walkthrough below.IntroductionCox is a lightweight, serverless framework for designing and managing
experiments. Inspired by our own struggles with ad-hoc filesystem-based
experiment collection, Cox aims
to be a minimal burden while inducing more organization. Created byLogan
EngstromandAndrew
Ilyas.Cox works by helping you easilylog,collect, andanalyzeexperimental results. For API documentation, seehere; below, we
provide a walkthrough that illustrates the most important features of Cox.Why "Cox"? (Aside): The name Cox draws both fromCoxswain, the person in charge of
steering the boat in a rowing crew, and from the name ofGertrude
Cox, a pioneer of experimental
design.InstallationCox can by installed via PyPI as:pip3installcoxCox requires Python 3 and has been tested with Python 3.7.Citation@unpublished{cox,
    title={Cox: A Lightweight Experimental Design Library},
    author={Logan Engstrom and Andrew Ilyas},
    year={2019},
    url={https://github.com/MadryLab/cox}
}Illustrative exampleimportosfromcox.storeimportStoreimportshutilimportsubprocessfromcox.readersimportCollectionReader"""Background: suppose we have two functions f(x, param) and g(x, param) that wewant to track as x ranges from 0 to 100, over a set of values for param. We alsowant to visualize f(x) with TensorBoard"""OUT_DIR=...POSSIBLE_PARAM_VALUES=[...]deff(x,param):...defg(x,param):...forparaminPOSSIBLE_PARAM_VALUES:# Creates a cox.Store, which stores a set of tables and a tensorboardstore=Store(OUT_DIR)# Create a table to store hyperparameters in for each runstore.add_table('metadata',{'param':float})# The metadata table will just have a single row with the param storedstore['metadata'].append_row({'param':param})# Create a table to store our resultsstore.add_table('results',{'f(x)':float,'g(x)':float})forxinrange(100):# Log f(x) to the results table and to tensorboardstore.log_table_and_tb('results',{'f(x)':f(x,param),})# Log g(x) to the table but not to TensorBoard. The working row has not# changed, so f(x) above and g(x) will be in the same rowstore['results'].update_row({'g(x)':g(x,param)})# Close the working rowstore['results'].flush_row()store.close()# Comparing results programmatically with CollectionReaderreader=CollectionReader(OUT_DIR)df=reader.df('results')m_df=reader.df('metadata')# Filter by experiments have "param" less than 1.0exp_ids=set(m_df[m_df['param']<1.0]['exp_id].tolist())print(df[df['exp_id'].isin(exp_ids)])# The filtered DataFrame# Finding which experiment has lowest minimum f(x)exp_id=df[df['results']==min(df['results'].tolist())]['exp_id'].tolist()[0]print(m_df[m_df['exp_id']==exp_id])# Metadata of the best experiment# Start tensorboard to compare across parameters who match the regex REGEXos.system("python -m cox.tensorboard_view --logdir OUT_DIR --format-str p-{param}\--filter-paramparamREGEX--metadata-tablemetadata"])Quick Logging OverviewThe cox logging system is designed for dealing with repeated experiments. The
user defines schemas forPandas
dataframesthat contain all the data necessary for each experiment instance. Each
experiment ran corresponds to adata store, and each specified dataframe
from above corresponds to a table within this store. The experiment stores are
organized within the same directory. Cox has a number of utilities for running
and collecting data from experiments of this nature.Interactive IntroductionWe use Cox most in our machine learning work, but Cox is agnostic to the type or
style of code that you write. To illustrate this, we go through an extremely
simple example in a walkthrough.Walkthrough 1: Logging in CoxNote 1: you can view all of the components of this running example in theexample file here!Note 2: a copy of this walkthrough is also available together with our API
documentation,hereIn this walkthrough, we'll be starting with the following simple piece of code,
which tries to finds the minimum of a quadratic function:importsysdeff(x):return(x-2.03)**2+3x=...tol=...step=...for_inrange(1000):# Take a uniform step in the direction of decreaseiff(x+step)<f(x-step):x+=stepelse:x-=step# If the difference between the directions# is less than the tolerance, stopiff(x+step)-f(x-step)<tol:breakInitializing storesLogging in Cox is done through theStoreclass, which can be created as follows:fromcox.storeimportStore# rest of program here...store=Store(OUT_DIR)Upon construction, theStoreinstance creates a directory with a randomuuidgenerated name inOUT_DIR, aHDFStorefor storing data, some logging
files, and a tensorboard directory (namedtensorboard). Therefore, after we run this command, ourOUT_DIRdirectory should look something like this:$lsOUT_DIR
7753a944-568d-4cc2-9bb2-9019cc0b3f49
$ls7753a944-568d-4cc2-9bb2-9019cc0b3f49
savestore.h5tensorboardThe experiment ID string7753a944-568d-4cc2-9bb2-9019cc0b3f49was
autogenerated. If we wanted to name the experiment something else, we could pass
it as the second parameter; i.e. making a store withStore(OUT_DIR, 'exp1')would make the corresponding experiment IDexp1.Creating tablesThe next step is to declare the data we want to store viatables. We can add
arbitrary tables according to our needs, but we need to specify the structure
ahead of time by passing the schema. In our case, we will start out with just a
simple metadata table containing the parameters used to run an instance of the
program above, along with a table for writing the result:store.add_table('metadata',{'step_size':float,'tolerance':float,'initial_x':float,'out_dir':str})store.add_table('result',{'final_x':float,'final_opt':float})Each table corresponds exactly to aPandas dataframefound in anHDFStoreobject.Note on serializationCox supports basic object types (likefloat,int,str, etc) along with any
kind of serializable object (viadillor using PyTorch's serialization
method). In particular, if we want to serialize an object we can pass one of the
following types:cox.store.[OBJECT|PICKLE|PYTORCH_STATE]as the type value
that is mapped to in the schema dictionary.cox.store.PYTORCH_STATEis
particularly useful for dealing with PyTorch objects like weights.
In detail:OBJECTcorresponds to storing the object as a
serialized string in the table,PICKLEcorresponds to storing the object as a
serialized string on disk in a separate file, andPYTORCH_STATEcorresponds to
storing the object as a serialized string on disk usingtorch.save.LoggingNow that we have a table, we can write rows to it! Logging in Cox is done in a
row-by-row manner: at any time, there is aworking rowthat can be appended
to/updated; the row can then be flushed (i.e. written to the file), which starts
a new (empty) working row. The relevant commands are:# This updates the working row, but does not write it permenantly yet!store['result'].update_row({"final_x":3.0})# This updates it againstore['result'].update_row({"final_opt":3.9409})# Write the row permenantly, and start a new working row!store['result'].flush_row()# A shortcut for appending a row directlystore['metadata'].append_row({'step_size':0.01,'tolerance':1e-6,'initial_x':1.0,'out_dir':'/tmp/'})Incremental updates withupdate_rowSubsequent calls to update_row will edit the same working row.
This is useful if different parts of the row are computed in different
functions/locations in the code, as it removes the need for passing statistics
around all over the place.Reading dataBy populating tables rows, we are really just adding rows to an underlyingHDFStoretable. If we want to read the store later, we can simply open another
store at the same location, and then read dataframes with simple commands:# Note that EXP_ID is the directory the store wrote to in OUT_DIRs=Store(OUT_DIR,EXP_ID)# Read tables we wrote earliermetadata=s['metadata'].dfresult=s['result'].dfprint(result)Inspecting theresulttable, we see the expected result in our Pandas dataframe!final_x   final_opt
0   3.000000   3.940900CollectionReader: Reading many experiments at onceNow, in our quadratic example, we aren't just going to try one set of
parameters, we are going to try a number of different values forstep_size,tolerance, andinitial_x, as we have not yet discovered convex optimization.
To do this, we just run the script above a bunch of times with the desired
hyperparameters,  supplying thesameOUT_DIRfor all of the runs (recall
thatcoxwill automatically create different,uuid-named folders insideOUT_DIRfor each experiment).Imagine that we have done so (using any standard tool, e.g.sbatchin SLURM,sklearngrid search, etc.), and that we have a directory full of stores (this
is why we useuuids instead of handpicked names!):$ls$OUT_DIRdrwxr-xr-x6engstrom0424807a-c9c0-4974-b881-f927fc5ae7c3
...
...
drwxr-xr-x6engstrome3646fcf-569b-46fc-aba5-1e9734fedbcf
drwxr-xr-x6engstromf23d6da4-e3f9-48af-aa49-82f5c017e14fNow, we want to collect all the results from this directory. We can usecox.readers.CollectionReaderto read all the tables together in a concatenatedpandastable.fromcox.readersimportCollectionReaderreader=CollectionReader(OUT_DIR)print(reader.df('result'))Which gives us all theresulttables concatenated together as aPandas DataFramefor easy manipulation:final_x   final_opt                                exp_id
0   1.000000    4.060900  ed892c4f-069f-4a6d-9775-be8fdfce4713
0   0.000010    7.120859  44ea3334-d2b4-47fe-830c-2d13dc0e7aaa
...
...
0   2.000000    3.000900  f031fc42-8788-4876-8c96-2c1237ceb63d
0 -14.000000  259.960900  73181d27-2928-48ec-9ac6-744837616c4bpandashas a ton of powerful utilities for searching through and
manipulating DataFrames. We recommend looking attheir
docsfor
information on how to do this. For convenience, we've given a few simple
examples below:df=reader.df('result')m_df=reader.df('metadata')# Filter by experiments have step_size less than 1.0exp_ids=set(m_df[m_df['step_size']<1.0]['exp_id].tolist())print(df[df['exp_id'].isin(exp_ids)])# The filtered DataFrame# Finding which experiment has the lowest final_optexp_id=df[df['final_opt']==min(df['final_opt'].tolist())]['exp_id'].tolist()[0]print(m_df[m_df['exp_id']==exp_id])# Metadata of the best experimentWalkthrough 2: UsingcoxwithtensorboardXNote: As with the first walkthrough, a working example file with all of
these commands can be foundhereHere, we'll show how to usecoxandtensorboardXin unison for logging.
We'll use the following simple running example:fromcox.storeimportStoreforslopeinrange(5):s=Store(OUT_DIR)# Create OUT_DIR/RANDOM_UUIDs.add_table('line_graphs',{'mx':int,'mx^2':int})s.add_table('metadata',{'slope':int})s['metadata'].append_row({'slope':slope})# GOAL: plot and log the lines "y=slope*x" and "y=slope*x^2"As previously mentioned,cox.Storeobjects also automatically creates atensorboardfolder that is written to via thetensorboardXlibrary. A createdcox.store.Storeobject will actually awriterproperty
that is a fully functioningSummaryWriterobject. That means we can plot the lines we want in TensorBoard as follows:forxinrange(10):s.writer.add_scalar('line',slope*x,x)s.writer.add_scalar('parabola',slope*(x**2),x)Unfortunately, TensorBoard data is quite hard to read/manipulate through means
other than the TensorBoard interface. For convenience, thestoreobject also
provides the ability to write to a table and thetensorboardXwriter at the same
time through thelog_table_and_tbfunction, meaning that we can replace the
above with:# Does the same thing as the example above but also stores the results in a# readable 'line_graphs' tableforxinrange(10):s.log_table_and_tb('line_graphs',{'mx':slope*x,'mx^2':slope*(x**2)})s['line_graphs'].flush_row()Viewing multiple tensorboards withcox.tensorboard_viewNote: thepython -m cox.tensorboard_viewcommand can be called ascox-tensorboardfrom the command lineContinuing with our running example, we may now want to visually compare
TensorBoards across multiple parameter settings. Fortunately,coxprovides utilities for comparing TensorBoards across experiments in a readable
way.  In our example, where we made aStoreobject and a table
calledmetadatawhere we stored hyperparameters. We also showed how to
integrate TensorBoard logging viatensorboardX. We'll now use thecox.tensorboard-viewutility to view the tensorboards from multiple jobs at
once (this is useful when comparing parameters for a grid search).The way to achieve this is through thecox.tensorboard_viewcommand, which is
called aspython3 -m cox.tensorboard_viewwith the following arguments:--logdir:(required), the directory where all of the stores are located--port:(default 6006), the port on which to run the tensorboard server--metadata-table(default "metadata"), the name of the table where the
hyperparameters are saved (i.e. "metadata" in our running example). This
should be a table with a single row, as in our running example.--filter-param(optional)Can be used more than once, filters out stores
from the tensorboard aggregation. For each argument of the form--filter-param PARAM_NAME PARAM_REGEX, only the stores wherePARAM_NAMEin
the metadata matchesPARAM_REGEXwill be kept.--format-str(required)How to display the name of the stores. Recall
that each store has auuid-generated name by default. This argument
determines how their names will be displayed in the TensorBoard. Curly braces
represent parameter values, and the uuid will always be appended to the name. So
in our running example,--format-str ss-{step_size}will result in a
TensorBoard with names of the formss-1.0-ed892c4f-069f-4a6d-9775-be8fdfce4713.So in our running example, if we run the following command, displaying the slope
in the TensorBoard names and filtering for slopes between 1 and 3:python3-mcox.tensorboard_view--logdirOUT_DIR--format-strslope-{slope}\--filter-paramslope[1-3]--metadata-tablemetadataorcox-tensorboard--logdirOUT_DIR--format-strslope-{slope}\--filter-paramslope[1-3]--metadata-tablemetadatathen navigating tolocalhost:6006yields: