Package: dnn-cool
dnn_cool: Deep Neural Networks for Conditional objective oriented learningWARNING: API is not yet stable, expect breaking changes in 0.x versions!To install, just do:pipinstalldnn_coolIntroduction: What isdnn_coolin a nutshell?Examples: a simple step-by-step example.Features: a list of the utilities thatdnn_coolprovides for youCustomization: Learn how to add new tasks, modify them, etc.Inspiration: list of papers and videos which inspired this libraryTo see the predefined tasks for this release, seelist of predefined tasksIntroductionA framework for multi-task learning in Pytorch, where you may precondition tasks and compose them into bigger tasks.
Many complex neural networks can be trivially implemented withdnn_cool.
For example, creating a neural network that does classification and localization is as simple as:@project.add_flowdeflocalize_flow(flow,x,out):out+=flow.obj_exists(x.features)out+=flow.obj_x(x.features)|out.obj_existsout+=flow.obj_y(x.features)|out.obj_existsout+=flow.obj_w(x.features)|out.obj_existsout+=flow.obj_h(x.features)|out.obj_existsout+=flow.obj_class(x.features)|out.obj_existsreturnoutIf for example you want to classify first if the camera is blocked and then do localizationgiven that the camera
is not blocked, you could do:@project.add_flowdeffull_flow(flow,x,out):out+=flow.camera_blocked(x.cam_features)out+=flow.localize_flow(x.localization_features)|(~out.camera_blocked)returnoutBased on these "task flows" as we call them,dnn_coolprovides a bunch offeatures.
Currently, this is the list of the predefined tasks (they are all located indnn_cool.task_flow):List of predefined tasksIn the current release, the following tasks are available out of the box:BinaryClassificationTask- sigmoid activation, thresholding decoder, binary cross entropy loss function. In the
examples above,camera_blockedandobj_existsareBinaryClassificationTasks.ClassificationTask- softmax activation, sorting classes decoder, categorical cross entropy loss. In the example
above,obj_classis aClassificationTaskMultilabelClassificationTask- sigmoid activation, thresholding decoder, binary cross entropy loss function.BoundedRegressionTask- sigmoid activation, rescaling decoder, mean squared error loss function. In the examples
above,obj_x,obj_y,obj_w,obj_hare bounded regression tasks.MaskedLanguageModelingTask- softmax activation, sorting decoder, cross entropy per token loss.TaskFlow- a composite task, that contains a list of children tasks. We saw 2 task flows above.ExamplesQuick Imagenet exampleWe just have to add aClassificationTasknamedclassifierand add the flow below:@project.add_flow()defimagenet_model(flow,x,out):out+=flow.classifier(x.features)returnoutThat's great! But what if there is not an object always? Then we have to first check if an object exists. Let's
add aBinaryClassificationTaskand use it as a precondition to classifier.@project.add_flow()defimagenet_model(flow,x,out):out+=flow.object_exists(x.features)out+=flow.classifier(x.features)|out.object_existsreturnoutBut what if we also want to localize the object? Then we have to add new tasks that regress the bounding box. Let's
call themobject_x,object_y,object_w,object_hand make them aBoundedRegressionTask. To avoid
preconditioning all tasks onobject_exists, let's group them first. Then we modify the
flow:@project.add_flow()defobject_flow(flow,x,out):out+=flow.classifier(x.features)out+=flow.object_x(x.features)out+=flow.object_y(x.features)out+=flow.object_w(x.features)out+=flow.object_h(x.features)returnout@project.add_flow()defimagenet_flow(flow,x,out):out+=flow.object_exists(x.features)out+=flow.object_flow(x.features)|out.object_existsreturnoutBut what if the camera is blocked? Then there is no need to do anything, so let's create a new flow
that executes ourimagenet_flowonly when the camera is not blocked.deffull_flow(flow,x,out):out+=flow.camera_blocked(x.features)out+=flow.imagenet_flow(x.features)|(~out.camera_blocked)returnoutBut what if for example we want to check if the object is a kite, and if it is, to classify its color?
Then we would have to modify ourobject_flowas follows:@project.add_flow()defobject_flow(flow,x,out):out+=flow.classifier(x.features)out+=flow.object_x(x.features)out+=flow.object_y(x.features)out+=flow.object_w(x.features)out+=flow.object_h(x.features)out+=flow.is_kite(x.features)out+=flow.color(x.features)|out.is_kitereturnoutI think you can see whatdnn_coolis meant to do! :)To see a full walkthrough on a synthetic dataset, check out theColab notebookor themarkdown write-up.FeaturesMain features are:Task preconditionMissing values handlingTask compositionTensorboard metrics loggingTask interpretationsTask evaluationTask threshold tuningDataset generationTree explanationsMemory balancing for dataparallelTask preconditioningUse the|for task preconditioning (think ofP(A|B)notation). Preconditioning -A | Bmeans that:Include the ground truth forBin the input batch when trainingWhen training, update the weights of theAonly whenBis satisfied in the ground truth.When training, compute the loss function forAonly whenBis satisfied in the ground truthWhen training, compute the metrics forAonly whenBis satisfied in the ground truth.When tuning threshold forA, optimize only on values for whichBis satisfied in the ground truth.When doing inference, compute the metrics forAonly when the precondition is satisfied according to the decoded
result of theBtaskWhen generating tree explanation in inference mode, do not show the branch forAifBis not
satisfied.When computing results interpretation, include only loss terms when the precondition is satisfied.Usually, you have to keep track of all this stuff manually, which makes adding new preconditions very difficult.dnn_coolmakes this stuff easy, so that you can chain a long list of preconditions without worrying you forgot
something.Missing valuesSometimes for an input you don't have labels for all tasks. Withdnn_cool, you can just mark the missing label anddnn_coolwill update only the weights of the tasks for which labels are available.This feature has the awesome property that you don't need a single dataset with all tasks labeled, you can
have different datasets for different tasks and it will work. For example, you can train a single object detection
neural network that trains its classifier head on ImageNet, and its detection head on COCO.Task compositionYou can group tasks in a task flow (we already saw 2 above -localize_flowandfull_flow). You can use this to
organize things better, for example when you want to precondition a whole task flow. For example:@project.add_flowdefface_regression(flow,x,out):out+=flow.face_x1(x.face_localization)out+=flow.face_y1(x.face_localization)out+=flow.face_w(x.face_localization)out+=flow.face_h(x.face_localization)out+=flow.facial_characteristics(x.features)returnoutTensorboard loggingdnn_coollogs the metrics per task in Tensorboard, e.g:Task interpretationAlso, the best and worst inputs per task are logged in the Tensorboard, for example if the input is an image:Task evaluationPer-task evaluation information is available, to pinpoint the exact problem in your network. An example
evaluation dataframe:task_pathmetric_namemetric_resnum_samples0camera_blockedaccuracy0.9803269961camera_blockedf1_score0.9743689962camera_blockedprecision0.9466359963camera_blockedrecall0.9601079964door_openaccuracy0.9212159025door_openf1_score0.9668599026door_openprecision0.9767499027door_openrecall0.9390389028door_lockedaccuracy0.9830392019door_lockedf1_score0.94837220110door_lockedprecision0.98258320111door_lockedrecall0.93478820112person_presentaccuracy0.99916670113person_presentf1_score0.93754170114person_presentprecision0.92733770115person_presentrecall0.96342870116person_regression.face_regression.face_x1mean_absolute_error0.013729261117person_regression.face_regression.face_y1mean_absolute_error0.023276161118person_regression.face_regression.face_wmean_absolute_error0.0074050361119person_regression.face_regression.face_hmean_absolute_error0.010161120person_regression.face_regression.facial_characteristicsaccuracy0.93262461121person_regression.body_regression.body_x1mean_absolute_error0.0083078561122person_regression.body_regression.body_y1mean_absolute_error0.015123461123person_regression.body_regression.body_wmean_absolute_error0.013021461124person_regression.body_regression.body_hmean_absolute_error0.010161125person_regression.body_regression.shirt_typeaccuracy_10.97993461126person_regression.body_regression.shirt_typeaccuracy_30.99333461127person_regression.body_regression.shirt_typeaccuracy_50.99052661128person_regression.body_regression.shirt_typef1_score0.92851661129person_regression.body_regression.shirt_typeprecision0.95982661130person_regression.body_regression.shirt_typerecall0.968146611Task threshold tuningMany tasks need to tune their threshold. Just callflow.get_decoder().tune()and you will get optimized thresholds
for the metric you define.Dataset generationAs noted above,dnn_coolwill automatically trace the tasks used as a precondition and include the ground truth for
them under the keygt.Tree explanationsExamples:├── inp 1
│   └── camera_blocked | decoded: [False], activated: [0.], logits: [-117.757324]
│       └── door_open | decoded: [ True], activated: [1.], logits: [41.11258]
│           └── person_present | decoded: [ True], activated: [1.], logits: [60.38873]
│               └── person_regression
│                   ├── body_regression
│                   │   ├── body_h | decoded: [29.672623], activated: [0.46363473], logits: [-0.14571853]
│                   │   ├── body_w | decoded: [12.86382], activated: [0.20099719], logits: [-1.3800735]
│                   │   ├── body_x1 | decoded: [21.34288], activated: [0.3334825], logits: [-0.69247603]
│                   │   ├── body_y1 | decoded: [18.468979], activated: [0.2885778], logits: [-0.9023013]
│                   │   └── shirt_type | decoded: [6 1 0 4 2 5 3], activated: [4.1331367e-23 3.5493638e-17 3.1328378e-26 5.6903808e-30 2.4471377e-25
 2.8071076e-29 1.0000000e+00], logits: [-20.549513  -6.88627  -27.734364 -36.34787  -25.6788   -34.751904
  30.990908]
│                   └── face_regression
│                       ├── face_h | decoded: [11.265154], activated: [0.17601803], logits: [-1.5435623]
│                       ├── face_w | decoded: [12.225838], activated: [0.19102871], logits: [-1.4433397]
│                       ├── face_x1 | decoded: [21.98834], activated: [0.34356782], logits: [-0.64743483]
│                       ├── face_y1 | decoded: [3.2855165], activated: [0.0513362], logits: [-2.9166584]
│                       └── facial_characteristics | decoded: [ True False  True], activated: [9.9999940e-01 1.2074912e-12 9.9999833e-01], logits: [ 14.240071 -27.442476  13.27557 ]but if the model thinks the camera is blocked, then the explanation would be:├── inp 2
│   └── camera_blocked | decoded: [ True], activated: [1.], logits: [76.367676]Memory balancingWhen usingnn.DataParallel, the computation of
the loss function is done on the main GPU, which leads to dramatically unbalanced memory usage if your outputs are big
and you have a lot of metrics (e.g segmentation masks, language modeling, etc).dnn_coolgives you a
convenient way to balance the memory in such situations - just a singlebalance_dataparallel_memory = Truehandles
this case for you by first reducing all metrics on their respective device, and then additionally aggregating
the results that were reduced on each device automatically. Here's an example memory usage:Before:After:CustomizationSinceflow.torch()returns a normalnn.Module, you can use any library you are used to. If you useCatalyst,dnn_coolprovides a bunch of useful callbacks. Creating
a new task is as simple as creating a new instance of this dataclass:@dataclassclassTask(ITask):name:strlabels:Anyloss:nn.Moduleper_sample_loss:nn.Moduleavailable_func:Callableinputs:Anyactivation:Optional[nn.Module]decoder:Decodermodule:nn.Modulemetrics:Tuple[str,TorchMetric]Alternatively, you can subclassITaskand implement its inferface.InspirationAndrej Karpathy: Tesla Autopilot and Multi-Task Learning for Perception and PredictionPyTorch at Tesla - Andrej Karpathy, TeslaMultitask learning - Andrew Ng