internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: fonttools/fonttools

Issue:
[varLib.instancer] axis=drop is broken
`git bisect` tells me that since 969d09dcafd3669753e30060ccca7e5fd4c356cd, `varLib.instancer wdth=drop` produces this error:

```
  File "/Users/simon/others-repos/fonttools/Lib/fontTools/varLib/instancer/__init__.py", line 1551, in main
    isFullInstance = {
  File "/Users/simon/others-repos/fonttools/Lib/fontTools/varLib/instancer/__init__.py", line 1552, in <setcomp>
    axisTag for axisTag, limit in axisLimits.items() if limit[0] == limit[2]
TypeError: 'NoneType' object is not subscriptable
```


Discussion:

Resolving patch:
diff --git a/Lib/fontTools/varLib/instancer/__init__.py b/Lib/fontTools/varLib/instancer/__init__.py
--- a/Lib/fontTools/varLib/instancer/__init__.py
+++ b/Lib/fontTools/varLib/instancer/__init__.py
@@ -1880,7 +1880,7 @@ def main(args=None):
     )
 
     isFullInstance = {
-        axisTag for axisTag, limit in axisLimits.items() if limit[0] == limit[2]
+        axisTag for axisTag, limit in axisLimits.items() if not isinstance(limit, tuple)
     }.issuperset(axis.axisTag for axis in varfont["fvar"].axes)
 
     instantiateVariableFont(


Test patch:
Repository: mosaicml/composer

Issue:
Configure Jenkins
Enable github actions for:

- [x] pytest CPU runner
- [x] formatting and type checking (`yapf`, `pyright`)
- [ ] docker builds
- [x] docs builds


Discussion:

Resolving patch:
diff --git a/composer/callbacks/profiler.py b/composer/callbacks/profiler.py
--- a/composer/callbacks/profiler.py
+++ b/composer/callbacks/profiler.py
@@ -1,3 +1,5 @@
+# Copyright 2021 MosaicML. All Rights Reserved.
+
 from __future__ import annotations
 
 import atexit
diff --git a/composer/callbacks/timing_monitor.py b/composer/callbacks/timing_monitor.py
--- a/composer/callbacks/timing_monitor.py
+++ b/composer/callbacks/timing_monitor.py
@@ -1,3 +1,5 @@
+# Copyright 2021 MosaicML. All Rights Reserved.
+
 from __future__ import annotations
 
 import logging
diff --git a/composer/trainer/entrypoint.py b/composer/trainer/entrypoint.py
--- a/composer/trainer/entrypoint.py
+++ b/composer/trainer/entrypoint.py
@@ -1,3 +1,5 @@
+# Copyright 2021 MosaicML. All Rights Reserved.
+
 import logging
 
 import composer


Test patch:
Repository: mosaicml/composer

Issue:
Add Colab Example
* Add Example Jupyter notebook to the examples folder
* Add "Open in Colab" to the README.md



Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -49,6 +49,7 @@ def package_files(directory):
     'sphinxcontrib.katex>=0.8.6',
     'sphinxext.opengraph>=0.4.2',
     'sphinx_rtd_theme>=1.0.0',
+    'testbook>=0.4.2',
     'myst-parser>=0.15.2',
 ]
 extra_deps['wandb'] = ['wandb>=0.12.2']


Test patch:
diff --git a/tests/examples/__init__.py b/tests/examples/__init__.py
new file mode 100644
--- /dev/null
+++ b/tests/examples/__init__.py
@@ -0,0 +1 @@
+# Copyright 2021 MosaicML. All Rights Reserved.
diff --git a/tests/examples/test_composer_ipynb.py b/tests/examples/test_composer_ipynb.py
new file mode 100644
--- /dev/null
+++ b/tests/examples/test_composer_ipynb.py
@@ -0,0 +1,22 @@
+# Copyright 2021 MosaicML. All Rights Reserved.
+
+import os
+
+import pytest
+import testbook
+import testbook.client
+
+import composer
+
+examples_path = os.path.join(os.path.dirname(composer.__file__), '..', 'examples')
+
+
+@testbook.testbook(os.path.join(examples_path, 'composer.ipynb'))
+@pytest.mark.timeout(120)  # long timeout to download the dataset (if needed) and train one epoch
+def test_composer_notebook(tb: testbook.client.TestbookNotebookClient):
+    tb.execute_cell("imports")
+    tb.execute_cell("hparams")
+    tb.inject("trainer_hparams.max_epochs = 1")
+    tb.execute_cell("trainer")
+    assert tb.get('mosaic_trainer').state.max_epochs == 1
+    tb.execute_cell("train")
Repository: mosaicml/composer

Issue:
Checkpointing fails when using the composer command
## Environment 
Running on colo on branch `moin/bert` rebased off of the latest dev: commit hash `be411d3f71feffcbab84da8671dd530fdbbc2f30`.

## To reproduce

Steps to reproduce the behavior:

1. Checkout `moin/bert` branch
2. Run `composer -n 8 examples/run_mosaic_trainer.py -f composer/yamls/models/glue/sst-2_checkpointing.yaml`

## Expected behavior

If I run `python examples/run_mosaic_trainer.py -f composer/yamls/models/glue/sst-2_checkpointing.yaml` instead of `composer -n 8`, then checkpointing works as expected, and we are able to save a checkpoint successfully and continue training. However, when I run it with `composer -n 8`, we see the following crash (exactly when checkpointing happens): 

```
[BATCH][step=599]: { "lr-AdamW/group0": 0.0000, }
Error in subprocess
----------Subprocess STDOUT----------

----------Subprocess STDERR----------

Command '['/usr/bin/python3.8', '-u', 'examples/run_mosaic_trainer.py', '-f', 'composer/yamls/models/glue/sst-2_checkpointing.yaml']' returned non-zero exit status 1.
Killing subprocess 33667 with SIGTERM
Killing subprocess 33662 with SIGTERM
Killing subprocess 33661 with SIGTERM
Killing subprocess 33669 with SIGTERM
Killing subprocess 33659 with SIGTERM
Waiting 30 seconds for processes to terminate...
```

For reference, checkpointing is set to every 600 iterations. There is no printed output.

## Additional context
This is the next step in getting BERT pre-training working (saving the pre-trained model, and then fine-tuning it on GLUE tasks).



Discussion:

Resolving patch:
diff --git a/composer/trainer/checkpoint.py b/composer/trainer/checkpoint.py
--- a/composer/trainer/checkpoint.py
+++ b/composer/trainer/checkpoint.py
@@ -119,7 +119,6 @@ def save_checkpoint(self, state: State, seed: int, device: Device, config: Optio
         # to all processes on checkpoint resume.
         # This will be fixed by: https://github.com/mosaicml/composer/issues/12
         state_dict = {
-            'state': state.state_dict(),  # should be the same across all ranks. per-rank state not stored
             'rng': self._get_rng_state(device=device),  # stored across all ranks
             'seed': seed,
         }
@@ -128,6 +127,9 @@ def save_checkpoint(self, state: State, seed: int, device: Device, config: Optio
             # Need the check down here so all the DDP syncs will work for generating the checkpoint
             return
 
+        # we add the state only on rank 0 since other processes don't have loggers to serialize
+        state_dict['state'] = state.state_dict()  # should be the same across all ranks. per-rank state not stored
+
         if config:
             hparams_path = os.path.join(self.checkpoint_folder, "hparams.yaml")
             os.makedirs(self.checkpoint_folder, mode=0o775, exist_ok=True)


Test patch:
Repository: mosaicml/composer

Issue:
Add venv into docker image to enable editable `pip install`
When trying to install composer with `pip install -e .` from within the docker image, we are seeing this error:
```
Traceback (most recent call last):
  File "/usr/bin/composer", line 33, in <module>
    sys.exit(load_entry_point('mosaicml', 'console_scripts', 'composer')())
  File "/usr/bin/composer", line 22, in importlib_load_entry_point
    for entry_point in distribution(dist_name).entry_points
  File "/usr/lib/python3.8/importlib/metadata.py", line 445, in distribution
    return Distribution.from_name(distribution_name)
  File "/usr/lib/python3.8/importlib/metadata.py", line 169, in from_name
    raise PackageNotFoundError(name)
importlib.metadata.PackageNotFoundError: mosaicml
```
This seems to be remedied by running the `pip install` from within a virtualenv. Can we bake a virtualenv into the docker image as a workaround?


Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -1,12 +1,16 @@
 # Copyright 2021 MosaicML. All Rights Reserved.
 
 import os
+import site
 import sys
 import textwrap
 
 import setuptools
 from setuptools import setup
 
+# From https://github.com/pypa/pip/issues/7953#issuecomment-645133255
+site.ENABLE_USER_SITE = "--user" in sys.argv[1:]
+
 
 def package_files(directory: str):
     # from https://stackoverflow.com/a/36693250


Test patch:
Repository: mosaicml/composer

Issue:
ResNet56 default num_classes argument
## 🚀 Feature Request
The `num_classes` argument for [ResNet56_cifar10](https://github.com/mosaicml/composer/blob/main/composer/models/resnet56_cifar10/model.py) should have a default value `num_classes=10`.

## Motivation

It felt silly when writing a demo notebook to have to specify `num_classes=10` when calling `composer.models.CIFAR10_ResNet56(num_classes=10)`. The model has "cifar10" in its name, and even if it didn't, it's most common use is for cifar10.

## Implementation

Does it require any changes beyond the `__init__()` signature?


Discussion:
Soliciting @A-Jacobson @hanlint @coryMosaicML for thoughts
Yeah, this should be an easy one-liner. 

Resolving patch:
diff --git a/composer/models/resnet56_cifar10/model.py b/composer/models/resnet56_cifar10/model.py
--- a/composer/models/resnet56_cifar10/model.py
+++ b/composer/models/resnet56_cifar10/model.py
@@ -22,7 +22,7 @@ class CIFAR10_ResNet56(MosaicClassifier):
 
     def __init__(
         self,
-        num_classes: int,
+        num_classes: int = 10,
         initializers: Optional[List[Initializer]] = None,
     ) -> None:
         if initializers is None:


Test patch:
Repository: mosaicml/composer

Issue:
Checkpoints stored in run directory cannot have ":" in their name
By default, the run directory contains a timestamp with ":" in its name. If the run directory is uploaded to an object store, any checkpoints within it cannot be used by `CheckpointLoader` because it parses the `load_path` as a URL: https://github.com/mosaicml/composer/blob/dev/composer/trainer/checkpoint.py#L93-L99


Discussion:
It seems that there is a handy workaround for this for now. The `RunDirectoryUploader` has an argument `object_name_prefix`. This argument defaults to the "run directory" if not provided, which is the timestring that breaks things. If `object_name_prefix` is specified by the user, then the checkpoints can be loaded properly.
This will be fixed -- will remove colons from the time string before using that as a folder name. Though allowed, I don't think colons should be used in folder names....

Resolving patch:
diff --git a/composer/cli/launcher.py b/composer/cli/launcher.py
--- a/composer/cli/launcher.py
+++ b/composer/cli/launcher.py
@@ -60,7 +60,7 @@ def get_parser():
                         default=None,
                         help=textwrap.dedent("""\
                             Directory to store run artifcats. 
-                            Defaults to runs/{datetime.datetime.now().isoformat()}/""")),
+                            Defaults to runs/{timestamp}/""")),
     parser.add_argument("-m",
                         "--module_mode",
                         action="store_true",
@@ -110,7 +110,7 @@ def launch_processes(nproc: int, world_size: int, base_rank: int, node_rank: int
     processes = []
 
     if run_directory is None:
-        run_directory = os.path.join("runs", datetime.datetime.now().isoformat())
+        run_directory = os.path.join("runs", datetime.datetime.now().isoformat().replace(":", "-"))
     os.makedirs(run_directory, exist_ok=True)
 
     if master_port is None:
diff --git a/composer/utils/run_directory.py b/composer/utils/run_directory.py
--- a/composer/utils/run_directory.py
+++ b/composer/utils/run_directory.py
@@ -50,7 +50,7 @@
 
 _RUN_DIRECTORY_KEY = "COMPOSER_RUN_DIRECTORY"
 
-_start_time_str = datetime.datetime.now().isoformat()
+_start_time_str = datetime.datetime.now().isoformat().replace(":", "-")
 
 __all__ = [
     "get_node_run_directory",


Test patch:
Repository: mosaicml/composer

Issue:
Move `ComposerTrainer` to top-level imports
Our most heavily used objects should be easily importable from `composer` via:
```
from composer import Trainer, ComposerModel
```
rather than remember their submodule:

```
from composer.models import ComposerModel
```

Especially the last one, its tricky to remember whether its `models` or `model`


Discussion:

Resolving patch:
diff --git a/composer/__init__.py b/composer/__init__.py
--- a/composer/__init__.py
+++ b/composer/__init__.py
@@ -20,6 +20,7 @@
 from composer.core import Timer as Timer
 from composer.core import TimeUnit as TimeUnit
 from composer.core import types as types
+from composer.models import ComposerModel as ComposerModel
 from composer.trainer import Trainer as Trainer
 
 __version__ = "0.3.1"


Test patch:
Repository: mosaicml/composer

Issue:
Require `split_batch_fn` only for `grad_accum > 1`
For easy out-of-the-box use with custom datatypes, we should only require `split_batch_fn` if `grad_accum > 1`


Discussion:

Resolving patch:
diff --git a/composer/core/data_spec.py b/composer/core/data_spec.py
--- a/composer/core/data_spec.py
+++ b/composer/core/data_spec.py
@@ -107,6 +107,10 @@ def _default_device_transforms(self, batch: Batch):
         return batch
 
     def _default_split_batch(self, batch: Batch, num_microbatches: int) -> Sequence[Batch]:
+        if num_microbatches < 1:
+            raise ValueError("num_microbatches must be at least 1")
+        if num_microbatches == 1:
+            return [batch]
         if not isinstance(batch, Sequence):
             raise ValueError(f'split_fn requires batch be a tuple pair of tensors, got {type(batch)}')
         x, y = batch


Test patch:
Repository: mosaicml/composer

Issue:
Pin the pyright version
The environment variable `PYRIGHT_PYTHON_FORCE_VERSION` needs to be set when pyright is installed from pip. Ideally we can set this variable in the setup.py (e.g. `os.environ['PYRIGHT_PYTHON_FORCE_VERSION'] = '...'`); but if not, then it should be in the `.ci/Jenkinsfile`.


Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -66,10 +66,10 @@ def package_files(directory: str):
     "custom_inherit==2.3.2",
     'junitparser>=2.1.1',
     'coverage[toml]>=6.1.1',
-    'fasteners>=0.16.3',  # run_directory_uploader tests require fasteners
+    'fasteners==0.17.3',  # run_directory_uploader tests require fasteners
     'pytest>=7.0.0',
-    'toml>=0.10.2',
-    'yapf>=0.32.0',
+    'toml==0.10.2',
+    'yapf==0.32.0',
     'isort>=5.9.3',
     'ipython>=7.29.0',
     'ipykernel>=6.5.0',
@@ -77,38 +77,38 @@ def package_files(directory: str):
     'yamllint>=1.26.2',
     'pytest-timeout>=1.4.2',
     'pyright==1.1.224.post1',
-    'recommonmark>=0.7.1',
+    'recommonmark==0.7.1',
     'sphinx>=4.2.0',
-    'sphinx_copybutton>=0.4.0',
-    'sphinx_markdown_tables>=0.0.15',
-    'sphinx-argparse>=0.3.1',
-    'sphinxcontrib.katex>=0.8.6',
-    'sphinxext.opengraph>=0.4.2',
-    'sphinxemoji>=0.2.0',
+    'sphinx_copybutton==0.5.0',
+    'sphinx_markdown_tables==0.0.15',
+    'sphinx-argparse==0.3.1',
+    'sphinxcontrib.katex==0.8.6',
+    'sphinxext.opengraph==0.6.1',
+    'sphinxemoji==0.2.0',
     'furo>=2022.1.2',
-    'sphinx-copybutton>=0.4.0',
-    'testbook>=0.4.2',
+    'sphinx-copybutton==0.5.0',
+    'testbook==0.4.2',
     'myst-parser==0.16.1',
     'pylint>=2.12.2',
     'docformatter>=1.4',
-    'sphinx_panels>=0.6.0',
+    'sphinx_panels==0.6.0',
 ]
 
 extra_deps["deepspeed"] = [
-    'deepspeed>=0.5.5',
+    'deepspeed==0.5.10',
 ]
 
 extra_deps["wandb"] = [
-    'wandb>=0.12.2',
+    'wandb==0.12.10',
 ]
 
 extra_deps["unet"] = [
-    'monai>=0.7.0',
+    'monai==0.8.1',
     'scikit-learn>=1.0.1',
 ]
 
 extra_deps["timm"] = [
-    'timm>=0.5.4',
+    'timm==0.5.4',
 ]
 
 extra_deps["nlp"] = [


Test patch:
Repository: mosaicml/composer

Issue:
BlurPool has incorrect docstrings
BlurPool docstrings imply that parameters have default values, but I don't see those in the function signatures: https://github.com/mosaicml/composer/blob/dev/composer/algorithms/blurpool/blurpool.py#L96



Discussion:

Resolving patch:
diff --git a/composer/algorithms/blurpool/blurpool.py b/composer/algorithms/blurpool/blurpool.py
--- a/composer/algorithms/blurpool/blurpool.py
+++ b/composer/algorithms/blurpool/blurpool.py
@@ -93,7 +93,7 @@ class BlurPool(Algorithm):
             See :class:`.BlurConv2d` for further discussion. Default: ``True``.
     """
 
-    def __init__(self, replace_convs: bool, replace_maxpools: bool, blur_first: bool) -> None:
+    def __init__(self, replace_convs: bool = True, replace_maxpools: bool = True, blur_first: bool = True) -> None:
         self.replace_convs = replace_convs
         self.replace_maxpools = replace_maxpools
         self.blur_first = blur_first


Test patch:
Repository: mosaicml/composer

Issue:
tutorial url for CallbackHparams doesn't exist
The url pointed to in https://github.com/mosaicml/composer/blob/dev/composer/callbacks/callback_hparams.py#L35 doesn't exist: https://docs.mosaicml.com/en/latest/tutorials/adding_models_datasets.html#trainer-with-yahp

(I'd like to add a callback...)


Discussion:
@dlwh thanks for pointing this out! We'll put in a fix for that link. 

For now, you can view the page at an old version of our site: https://docs.mosaicml.com/en/v0.3.1/tutorials/adding_models_datasets.html#trainer-with-yahp

Resolving patch:
diff --git a/composer/callbacks/callback_hparams.py b/composer/callbacks/callback_hparams.py
--- a/composer/callbacks/callback_hparams.py
+++ b/composer/callbacks/callback_hparams.py
@@ -31,12 +31,7 @@
 
 @dataclass
 class CallbackHparams(hp.Hparams, abc.ABC):
-    """Base class for callback hyperparameters.
-
-    Callback parameters that are added to the callbacks argument of
-    :attr:`~composer.trainer.trainer_hparams.TrainerHparams` (e.g., via YAML or the CLI). See `Trainer with YAHP <https://docs.mosaicml.com/en/latest/tutorials/adding_models_datasets.html#trainer-with-yahp>`_ for more details.
-    These are initialized in the training loop.
-    """
+    """Base class for Callback hyperparameters."""
 
     @abc.abstractmethod
     def initialize_object(self) -> Callback:


Test patch:
Repository: mosaicml/composer

Issue:
Training freezes when using multi-gpu jobs with block-wise stochastic depth
## To reproduce

When I run the following command:
`composer -n 2 examples/run_composer_trainer.py -f composer/yamls/models/resnet50.yaml --algorithms stochastic_depth --algorithms.stochastic_depth.target_layer_name ResNetBottleneck --algorithms.stochastic_depth.drop_rate 0.0 --loggers tqdm`

The training loop freezes. Using only 1 GPU or switching to sample-wise stochastic depth (`--algorithms.stochastic_depth.stochastic_method sample`) seem to work as expected

Appears to be a bug with the block-wise stochastic depth layers, may be related to #296. Should probably be fixed before v0.4.



Discussion:
Been looking into this. Somehow, the code is hanging at [this line](https://github.com/pytorch/pytorch/blob/master/torch/cuda/amp/grad_scaler.py#L284) in PyTorch's native gradient scaler. `optimizer_state["found_inf_per_device"]` seems to correspond to some sort of voodoo-magic thing that causes a distributed communications call when accessed. It seems to be linked to the `_MultiDeviceReplicator` defined earlier in that file.

Debugging this by tracing looks like it's gonna get very difficult very quickly. I'll try another approach tomorrow?
Not sure if this helps, but the code freezes when `find_unused_parameters=True` even if no layers are dropped. If `find_unused_parameters=False` and layers are dropped, then it also freezes. Could there be issues with finding the unused parameters? Any possibility it is related manually adjusting the parameters in optimizer after surgery?

Resolving patch:
diff --git a/composer/trainer/trainer.py b/composer/trainer/trainer.py
--- a/composer/trainer/trainer.py
+++ b/composer/trainer/trainer.py
@@ -195,7 +195,7 @@ def _distribute_and_get_random_seed(seed: Optional[int], device: Device):
 def _get_ddp_sync_strategy(ddp_sync_strategy: Optional[Union[str, DDPSyncStrategy]], find_unused_parameters: bool):
     if ddp_sync_strategy is None:
         if find_unused_parameters:
-            ddp_sync_strategy = DDPSyncStrategy.FORCED_SYNC
+            ddp_sync_strategy = DDPSyncStrategy.MULTI_AUTO_SYNC
         else:
             ddp_sync_strategy = DDPSyncStrategy.SINGLE_AUTO_SYNC
     else:


Test patch:
Repository: mosaicml/composer

Issue:
MLPerf Callback does not use global batch size in dataset length calculation
The MLPerf callback release as part of Composer 0.7.1 calculates FFCV dataset length for device batch size only:

```
if isinstance(dataloader, ffcv.loader.Loader):
    return (dataloader.batch_size, len(dataloader) * dataloader.batch_size)  # type: ignore
```

This results in the MLPerf log printing a train and val length that is 8x off from actual:

```
:::MLLOG {"namespace": "", "time_ms": 1653030529909, "event_type": "POINT_IN_TIME", "key": "train_samples", "value": 160000, "metadata": {"file": "/usr/local/lib/python3.9/dist-packages/composer/callbacks/mlperf.py", "lineno": 235}}
:::MLLOG {"namespace": "", "time_ms": 1653030529909, "event_type": "POINT_IN_TIME", "key": "
```


Discussion:

Resolving patch:
diff --git a/composer/callbacks/mlperf.py b/composer/callbacks/mlperf.py
--- a/composer/callbacks/mlperf.py
+++ b/composer/callbacks/mlperf.py
@@ -250,7 +250,11 @@ def _get_dataloader_stats(self, dataloader: Iterable):
             import ffcv  # type: ignore
 
             if isinstance(dataloader, ffcv.loader.Loader):
-                return (dataloader.batch_size, len(dataloader) * dataloader.batch_size)  # type: ignore
+                # Use the cached attribute ffcv.init_traversal_order to compute number of samples
+                return (
+                    dataloader.batch_size,  # type: ignore
+                    len(dataloader.next_traversal_order()) * dist.get_world_size()  # type: ignore
+                )
         except ImportError:
             pass
 
@@ -274,8 +278,8 @@ def fit_start(self, state: State, logger: Logger) -> None:
                 constants.SEED: state.seed,
                 constants.GLOBAL_BATCH_SIZE: batch_size * dist.get_world_size(),
                 constants.GRADIENT_ACCUMULATION_STEPS: state.grad_accum,
-                constants.TRAIN_SAMPLES: num_samples * dist.get_world_size(),
-                constants.EVAL_SAMPLES: eval_num_samples * dist.get_world_size(),
+                constants.TRAIN_SAMPLES: num_samples,
+                constants.EVAL_SAMPLES: eval_num_samples,
             })
 
         if _local_rank_zero():


Test patch: