Package: bocas
Bocasbocasis an opinionated open source framework for organizing,
orchestrating, and ultimately publishing research experiments.Some design highlights ofbocasinclude:the ability to cache artifacts between experiment runsthe de-coupling of plot generation and training jobsbocasaugments theml-collectionslibrary to allow you to describes an
array of experiments in a single configrun all of the experiments with a single commandgather artifacts from the experimentsaggregate the results into plots, tables, and figures for use in your final reporteasily combine results from multiple experimentsand more!Quick-StartBasic: Oxford 102 flowers classification exampleIntermediate: Object detection benchmarks with KerasCVOverviewOverviewUsingbocasis easy!To get started, you need to be familiar with a few concepts.
This overview covers everything you need to know.To quickly jump right into things, check out the Oxford 102 flowers classification example.Tasks & TacticsIn the mental model ofbocasthere existsTasksandTactics.  ATaskis
something like: "classify images from MNIST", or "cluster samples into N classes", or
"perform generative learning in X style".ATacticrefers to the combination of all the details used to produce a
solution to a Task.  For example, one such Tactic for solving MNIST classification might
be to train a ResNet50V2 on data augmented with AugMix.Typically, to get a publishable result your paper will require you to have numerous
tactics to benchmark your novel tactic against.Typically, a research work will have many Tasks: where
the overall goal of the paper is to benchmark a new Tactic's ability at solving a variety
of tasks.bocasis structured around this idea: you will have at least one Task, and
each Task may be solved by numerous tactics.
As such, I recommend breaking your codebase down at theTasklevel, structuring your
paper's artifact with splits made on theTasklevel.  For example, a classification
paper might have the structure:- tasks/
      - mnist/
            - ...
      - imagenet/
            - ...Code Structurebocasprovides an opinionated framework for generatingKeeping these concepts in mind,bocasrecommends that you structure your code
into three levels:library/holds anything unique to your report/paper/publication.  This might include
a new augmentation, a newkeras.Layer, a new loss function, or a new metric.tasks/holds all of the tasks to benchmark your new technique on.paper/holds theLatexorMarkdowncode required to render your paperpaper/artifactssubdirectory ofpaperthat holds all of the artifacts produced by
thetasks.  Typically when running a Task sweep you'll want to provide this directory
to your scripts.Your tasks should be structured as follows:All code for a task should reside intasks/{task}/, i.e.tasks/oxford_102.
You should create arun.pyscript.  This script must have arun()method that
accepts anml_collections.ConfigDictas its first positional argument.  If you follow
the example in theOxford Flowers 102 example, yourrun.pyfile will support both independent run and mass-scale sweeps:defrun(config):name=f'{config.optimizer}'train_ds,test_ds=tfds.load("oxford102",as_supervised=True,split=["train","test"])model=keras_cv.models.ResNet50V2(include_rescaling=True,include_top=True,classes=102)model.compile(loss="mse",optimizer=config.optimizer)history=model.fit(train_ds,epochs=10)returnbocas.Result(name=name,artifacts=[bocas.artifacts.KerasHistory(history,name="fit_history"),],)Once you are happy with the results from a singlerun.pyrun, create asweep.pyconfig file.  Insweep.py, specify aml_collections.ConfigDictcontainingbocas.Sweepobjects for any value you'd like to sweep oer.config=ml_collections.ConfigDict()config.static_value='any-string-or-int-or-float-or-python-object'config.optimizer=bocas.Sweep(['sgd','adam'])Anytime a value of typebocas.Sweep()is encountered, the product of all
other definedbocas.Sweep()parameters is run with the addition of the new
values in that sweep.Be careful with this!  It is easy to create a lot of experiments:config=ml_collections.ConfigDict()config.learning_rate=bocas.Sweep([x/100forxinrange(5,21)])config.optimizer=bocas.Sweep(['sgd','adam'])config.model=bocas.Sweep(['resnet50','resnet50v2','densenet101','efficientnet'])This configuration already contains15 * 2 * 4or120runs!  That is probably
way more than you'd like.  Try to define a few experiments that are all encompassing.
To accomplish this, run hyper parameter sweeps separately, and hardcode the values into
the final runs that are used to produce the charts.After all of your runs are complete, create some charts and plots.  Save them to your
designated directory in yourpaper/directory so that they are rendered
into your updated paper.I recommend writing a script to produce desired plots based on the artifacts that can
be run entirely separately from your experiments themselves.  Any example of this can
be found in theoxford_102example:# scripts/create_plots.pyresults=bocas.Result.load_collection("artifacts/")metrics_to_plot={}forexperimentinresults:metrics=experiment.get_artifact("fit_history").metricsmetrics_to_plot[f"{experiment.name}Train"]=metrics["accuracy"]metrics_to_plot[f"{experiment.name}Validation"]=metrics["val_accuracy"]luketils.visualization.line_plot(metrics_to_plot,path=f"{paper_dir}/results/combined-accuracy.png",title="Model Accuracy",)Check out the full code in oxford_102.Conclusions & Further ReadingThats all it takes to get running withbocas.  Please check out theexamples/directory for more reading.  It contains a few more patterns
that might be useful in structuring your experiments.Limitations:warning: right nowbocasis under active development :warning:While the API is relatively straightforward and simple,bocaslacks support for multi-worker experiment runs.  This means that you will need to run
all of your experiments concurrently on a single machine.  If you are running 10-20fit()loops to convergence, this will likely be an extremely expensive process.Personally, I'd rather just wait for my experiments to run then fiddle with a ton of
infrastructure.  That being said, I mainly run small scale research.If someone wants to contribute distributed runs, feel free!LicenseApache v2 LicenseContributingContributions are more than welcome tobocas.Please see the GitHub issue tracker, and feel free to pick up any issue annotated
withContribution Welcome.Additionally, bug reports are not only welcome but encouraged.Help me improvebocas!I made this project because I needed the tool.
I'm sure many others do as well.Thanks!If you find this tool helpful, please toss a GitHub star on the repo and follow me on Twitter.Thank you to all of our GitHub contributors: