anon-iclr-submission/benchname-bug-localization huggingface dataset:
Repository: vantage6/vantage6
Issue: GET `role/{id}` doesn't give error if id doesn't exist

The following is not consistent in the API:

GET organization/999 -> `"msg": "Organization id=999 not found!"` (status 404)
GET user/999 -> `"msg": "user id=999 is not found"`  (status 404)
(same for other resources)

however, for `role` we find:

GET role/999 -> `{}` (empty dictionary, status 200)

We need to adjust the role endpoint to also yield an error instead of an empty dictionary

Changed files:
vantage6-server/vantage6/server/resource/role.py

diff --git a/vantage6-server/vantage6/server/resource/role.py b/vantage6-server/vantage6/server/resource/role.py
index 3971cf8b..5a052868 100644
--- a/vantage6-server/vantage6/server/resource/role.py
+++ b/vantage6-server/vantage6/server/resource/role.py
@@ -296,6 +296,10 @@ class Role(RoleBase):
     def get(self, id):
         role = db.Role.get(id)
 
+        if not role:
+            return {"msg": f"Role with id={id} not found."}, \\
+                HTTPStatus.NOT_FOUND
+
         # check permissions. A user can always view their own roles
         if not (self.r.v_glo.can() or role in g.user.roles):
             if not (self.r.v_org.can()
Repository: dandi/dandi-cli
Issue: ngff upload running into validation errors

i don't think there is a directory more the 5 levels into the ngff file. also, if there is, it would be good to know an example of one in the error.

```
$ dandi validate .
2022-04-25 09:34:22,018 [   ERROR] micr/sub-MITU01_ses-20210521h17m17s06_sample-178_stain-NPY_run-1_chunk-1_SPIM.ngff: 1 error(s)
2022-04-25 09:34:22,019 [   ERROR]   Zarr directory tree more than 5 directories deep
```

this error repeats for all ngff files in the directory


Changed files:
dandi/consts.py

diff --git a/dandi/consts.py b/dandi/consts.py
index a3506ce..ec5e944 100644
--- a/dandi/consts.py
+++ b/dandi/consts.py
@@ -75,7 +75,7 @@ PUBLISHED_VERSION_REGEX = r"[0-9]+\\.[0-9]+\\.[0-9]+"
 
 #: Regular expression for a valid Dandiset version identifier.  This regex is
 #: not anchored.
-VERSION_REGEX = fr"(?:{PUBLISHED_VERSION_REGEX}|draft)"
+VERSION_REGEX = rf"(?:{PUBLISHED_VERSION_REGEX}|draft)"
 
 
 class EmbargoStatus(Enum):
@@ -152,7 +152,7 @@ VIDEO_FILE_EXTENSIONS = [".mp4", ".avi", ".wmv", ".mov", ".flv", ".mkv"]
 VIDEO_FILE_MODULES = ["processing", "acquisition"]
 
 #: Maximum allowed depth of a Zarr directory tree
-MAX_ZARR_DEPTH = 5
+MAX_ZARR_DEPTH = 7
 
 #: MIME type assigned to & used to identify Zarr assets
 ZARR_MIME_TYPE = "application/x-zarr"
Repository: vantage6/vantage6
Issue: Error in GET `collaboration?organization_id=xxx`  for users without global collaboration view permissions

When I run the request above as a user that does not have global permission to view collaborations but only for their own organization, I get the message `An unexpected error occurred on the server!`.

In the server, there is a SQL error:
```
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) ambiguous column name: organization.id
[SQL: SELECT count(*) AS count_1
FROM (SELECT DISTINCT collaboration.id AS collaboration_id, collaboration.name AS collaboration_name, collaboration.encrypted AS collaboration_encrypted
FROM collaboration JOIN "Member" ON collaboration.id = "Member".collaboration_id JOIN organization ON organization.id = "Member".organization_id JOIN "Member" AS "Member_1" ON collaboration.id = "Member_1".collaboration_id JOIN organization ON organization.id = "Member_1".organization_id
WHERE organization.id = ? AND (EXISTS (SELECT 1
FROM "Member", organization
WHERE collaboration.id = "Member".collaboration_id AND organization.id = "Member".organization_id AND organization.id = ?))) AS anon_1]
[parameters: ('2', 2)]
```

Changed files:
vantage6-server/vantage6/server/resource/collaboration.py

diff --git a/vantage6-server/vantage6/server/resource/collaboration.py b/vantage6-server/vantage6/server/resource/collaboration.py
index 332cc618..9bceea97 100644
--- a/vantage6-server/vantage6/server/resource/collaboration.py
+++ b/vantage6-server/vantage6/server/resource/collaboration.py
@@ -202,8 +202,15 @@ class Collaborations(CollaborationBase):
 
         # find collaborations containing a specific organization
         if 'organization_id' in args:
-            q = q.join(db.Member).join(db.Organization)\\
-                 .filter(db.Organization.id == args['organization_id'])
+            if not self.r.v_glo.can() and \\
+                    args['organization_id'] != str(auth_org_id):
+                return {'msg': 'You lack the permission to request '
+                        'collaborations for this organization!'}
+            elif self.r.v_glo.can():
+                q = q.join(db.Member).join(db.Organization)\\
+                    .filter(db.Organization.id == args['organization_id'])
+            # else: no filter if user can only view collaborations of own
+            # organization: the arg 'organization_id' is then superfluous
 
         # filter based on permissions
         if not self.r.v_glo.can():
@@ -339,7 +346,7 @@ class Collaboration(CollaborationBase):
                 HTTPStatus.UNAUTHORIZED
 
         collaboration.delete()
-        return {"msg": f"node id={id} successfully deleted"}, \\
+        return {"msg": f"Collaboration id={id} successfully deleted"}, \\
             HTTPStatus.OK
Repository: vantage6/vantage6
Issue: [Bug] client.node.list filters by default on offline nodes

**Describe the bug**
`client.node.list` filters on offline nodes by default. It should not filter on the _status_ field at all.

**To Reproduce**
```
>>> client.node.list()
Making request: GET | http://127.0.0.1:5000/node | {'page': 1, 'per_page': 20, 'include': ['metadata'], 'name': None, 'organization_id': None, 'collaboration_id': None, 'ip': None, 'status': 'offline', 'last_seen_from': None, 'last_seen_till': None}
{'data': [], 'links': {'first': '/node?page=1&per_page=20&include=metadata&status=offline', 'self': '/node?page=1&per_page=20&include=metadata&status=offline'}}
```

Changed files:
vantage6-client/vantage6/client/__init__.py

diff --git a/vantage6-client/vantage6/client/__init__.py b/vantage6-client/vantage6/client/__init__.py
index 46bd8e3e..0f96d7f9 100644
--- a/vantage6-client/vantage6/client/__init__.py
+++ b/vantage6-client/vantage6/client/__init__.py
@@ -941,10 +941,11 @@ class UserClient(ClientBase):
                 'page': page, 'per_page': per_page, 'include': includes,
                 'name': name, 'organization_id': organization,
                 'collaboration_id': collaboration, 'ip': ip,
-                'status': 'online' if is_online else 'offline',
                 'last_seen_from': last_seen_from,
                 'last_seen_till': last_seen_till
             }
+            if is_online is not None:
+                params['status'] = 'online' if is_online else 'offline'
             return self.parent.request('node', params=params)
 
         @post_filtering(iterable=False)
Repository: vantage6/vantage6
Issue: [Bug] Empty values in PATCH `/user/<id>` are not propagated to database

**Describe the bug**
If we do a PATCH `/user/<id>` and update a field with empty value, that field is not updated in the database.
E.g. if a user has firstname 'Alice' and we try to edit that to '' (empty string), the firstname will not be updated.
Similar for other fields in the user endpoint.
We should also check whether this may be the case for other PATCH endpoints.

**Expected behavior**
If someone wants to put an empty string in a PATCH request, that should be done (given that it's allowed for that field)

**Details**
 - v6 version 3.3.7


Changed files:
vantage6-server/vantage6/server/resource/user.py

diff --git a/vantage6-server/vantage6/server/resource/user.py b/vantage6-server/vantage6/server/resource/user.py
index b656a509..d945623e 100644
--- a/vantage6-server/vantage6/server/resource/user.py
+++ b/vantage6-server/vantage6/server/resource/user.py
@@ -556,19 +556,27 @@ class User(UserBase):
             return {"msg": "You cannot change your password here!"}, \\
                 HTTPStatus.BAD_REQUEST
 
-        if data["username"]:
-            if (user.username != data["username"] and
+        if data["username"] is not None:
+            if data["username"] == '':
+                return {
+                    "msg": "Empty username is not allowed!"
+                }, HTTPStatus.BAD_REQUEST
+            elif (user.username != data["username"] and
                     db.User.exists("username", data["username"])):
                 return {
                     "msg": "User with that username already exists"
                 }, HTTPStatus.BAD_REQUEST
             user.username = data["username"]
-        if data["firstname"]:
+        if data["firstname"] is not None:
             user.firstname = data["firstname"]
-        if data["lastname"]:
+        if data["lastname"] is not None:
             user.lastname = data["lastname"]
-        if data["email"]:
-            if (user.email != data["email"] and
+        if data["email"] is not None:
+            if data["email"] == '':
+                return {
+                    "msg": "Empty email is not allowed!"
+                }, HTTPStatus.BAD_REQUEST
+            elif (user.email != data["email"] and
                     db.User.exists("email", data["email"])):
                 return {
                     "msg": "User with that email already exists."
Repository: vantage6/vantage6
Issue: [Bug] Error in `client.util.change_my_password`

**Describe the bug**
Reported by Stefan Wolfsheimer, thanks Stefan!

`client.util.change_my_password` yields error:


```
DEBUG:urllib3.connectionpool:http://localhost:5000 "POST /api/recover/change HTTP/1.1" 404 232
Traceback (most recent call last):
  File "set_root_password.py", line 44, in <module>
    client.util.change_my_password(ORIG_ROOT_PASSWORD, ROOT_PASSWORD)
  File "/opt/vantage6/lib/python3.8/site-packages/vantage6/client/__init__.py", line 684, in change_my_password
    result = self.parent.request(
  File "/opt/vantage6/lib/python3.8/site-packages/vantage6/client/__init__.py", line 212, in request
    return self.request(endpoint, json, method, params,
  File "/opt/vantage6/lib/python3.8/site-packages/vantage6/client/__init__.py", line 217, in request
    return response.json()
  File "/opt/vantage6/lib/python3.8/site-packages/requests/models.py", line 900, in json
    return complexjson.loads(self.text, **kwargs)
  File "/usr/lib/python3.8/json/__init__.py", line 357, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.8/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.8/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
```

**Expected behavior**
Change password without error


Changed files:
vantage6-client/vantage6/client/__init__.py

diff --git a/vantage6-client/vantage6/client/__init__.py b/vantage6-client/vantage6/client/__init__.py
index 46bd8e3e..c25d4a40 100644
--- a/vantage6-client/vantage6/client/__init__.py
+++ b/vantage6-client/vantage6/client/__init__.py
@@ -682,7 +682,7 @@ class UserClient(ClientBase):
                 Message from the server
             """
             result = self.parent.request(
-                'recover/change', method='post', json={
+                'password/change', method='patch', json={
                     'current_password': current_password,
                     'new_password': new_password
                 }
Repository: vantage6/vantage6
Issue: [Bug] starting container with vpn takes a lot of time

**Describe the bug**
Starting a container on a node that has vpn enabled takes about 3 minutes more than starting it on the same node with vpn disabled.

**To Reproduce**
Steps to reproduce the behavior:
With vpn_subnet in the node config, start the node, run a task. Use the portal UI to see start and finish time of the task.

Comment out the vpn_subnet line in the config. Repeat the above.
See that without the vpn configured the task takes about 3 minutes less.



Changed files:
vantage6-node/vantage6/node/docker/vpn_manager.py

diff --git a/vantage6-node/vantage6/node/docker/vpn_manager.py b/vantage6-node/vantage6/node/docker/vpn_manager.py
index cdadf289..a99f85b2 100644
--- a/vantage6-node/vantage6/node/docker/vpn_manager.py
+++ b/vantage6-node/vantage6/node/docker/vpn_manager.py
@@ -350,29 +350,37 @@ class VPNManager(DockerBaseManager):
             return None  # no port assigned if no VPN is available
 
         # Get IP Address of the algorithm container
+        self.log.debug("Getting IP address of algorithm container")
         algo_helper_container.reload()  # update attributes
         algo_ip = self.get_isolated_netw_ip(algo_helper_container)
 
         # Set ports at which algorithm containers receive traffic
+        self.log.debug("Finding exposed ports of algorithm container")
         ports = self._find_exposed_ports(algo_image_name)
 
         # Find ports on VPN container that are already occupied
         cmd = (
             'sh -c '
-            '"iptables -t nat -L PREROUTING | awk \\'{print $7}\\' | cut -c 5-"'
+            '"iptables -t nat -L PREROUTING -n | '
+            'awk \\'{print $7}\\' | cut -c 5-"'
         )
         occupied_ports = self.vpn_client_container.exec_run(cmd=cmd)
+
         occupied_ports = occupied_ports.output.decode('utf-8')
         occupied_ports = occupied_ports.split('\\n')
         occupied_ports = \\
             [int(port) for port in occupied_ports if port != '']
+        self.log.debug(f"Occupied ports: {occupied_ports}")
 
         # take first available port
         vpn_client_port_options = set(FREE_PORT_RANGE) - set(occupied_ports)
         for port in ports:
-            port['port'] = vpn_client_port_options.pop()
+            port_ = vpn_client_port_options.pop()
+            self.log.debug(f"Assigning port {port_} to algorithm port")
+            port['port'] = port_
 
         vpn_ip = self.get_vpn_ip()
+        self.log.debug(f"VPN IP: {vpn_ip}")
 
         # Set up forwarding VPN traffic to algorithm container
         command = 'sh -c "'
Repository: vantage6/vantage6
Issue: [Bug] `AlgorithmClient` object has no attribute `organization_id`

**Describe the bug**
This line requires `self.whoami` to be set, however this is only done when the authentication method is triggered. This does not happen for algorithm containers.

https://github.com/vantage6/vantage6/blame/7dd509c35f0cf19ca18e7d84462e3aee5a093100/vantage6-client/vantage6/client/algorithm_client.py#L49-L50

Changed files:
vantage6-client/vantage6/client/algorithm_client.py

diff --git a/vantage6-client/vantage6/client/algorithm_client.py b/vantage6-client/vantage6/client/algorithm_client.py
index 05f13673..3635a2d7 100644
--- a/vantage6-client/vantage6/client/algorithm_client.py
+++ b/vantage6-client/vantage6/client/algorithm_client.py
@@ -46,7 +46,7 @@ class AlgorithmClient(ClientBase):
         self.database = container_identity.get('database')
         self.host_node_id = container_identity.get("node_id")
         self.collaboration_id = container_identity.get("collaboration_id")
-        self.organization_id = self.whoami.organization_id
+        self.organization_id = container_identity.get("organization_id")
         self.log.info(
             f"Container in collaboration_id={self.collaboration_id} \\n"
             f"Key created by node_id {self.host_node_id} \\n"
Repository: vantage6/vantage6
Issue: Nodes appear to be offline

If the server reboots, the nodes status field is not set properly when using multiple workers. It seems likely that the `start()` method is triggered after some of the clients (=nodes) reconnected. 

Reported by @anjavangestel 

Changed files:
vantage6-server/vantage6/server/__init__.py

diff --git a/vantage6-server/vantage6/server/__init__.py b/vantage6-server/vantage6/server/__init__.py
index a525cd5c..65a305c6 100644
--- a/vantage6-server/vantage6/server/__init__.py
+++ b/vantage6-server/vantage6/server/__init__.py
@@ -78,6 +78,9 @@ class ServerApp:
         # Setup the Flask-Mail client
         self.mail = MailService(self.app, Mail(self.app))
 
+        # Set nodes to offline -- required before setting up the SocketIO
+        self._set_nodes_offline()
+
         # Setup websocket channel
         self.socketio = self.setup_socket_connection()
 
@@ -129,6 +132,14 @@ class ServerApp:
 
         return socketio
 
+    @staticmethod
+    def _set_nodes_offline():
+        """Set all nodes to offline (before starting the socket connection)."""
+        nodes = db.Node.get()
+        for node in nodes:
+            node.status = 'offline'
+            node.save()
+
     @staticmethod
     def configure_logging():
         """Turn 3rd party loggers off."""
@@ -407,14 +418,6 @@ class ServerApp:
                            password=SUPER_USER_INFO['password'])
             user.save()
 
-        # set all nodes to offline
-        # TODO: this is *not* the way
-        nodes = db.Node.get()
-        for node in nodes:
-            node.status = 'offline'
-            node.save()
-        # session.commit()
-
         return self
Repository: vantage6/vantage6
Issue: [Bug] `pull_if_newer` function broken on the new Harbor API

**Describe the bug**
`pull_if_newer` method is broken (probably since the upgrade of our Harbor instance). This prevents nodes to pull the latest images. (The node did pull if no local image is present)

**To Reproduce**
1. Update a remote (algorithm/node) image, make sure you have an older version locally (but ofc. not the new one)
2. Start the node or algorithm
3. Check the version of the container that is used

**Expected behavior**
pull the image if the remote is newer


Changed files:
vantage6/vantage6/cli/server.py, vantage6-common/vantage6/common/docker/addons.py, vantage6/vantage6/cli/node.py

diff --git a/vantage6-common/vantage6/common/docker/addons.py b/vantage6-common/vantage6/common/docker/addons.py
index a7572528..7bb0271c 100644
--- a/vantage6-common/vantage6/common/docker/addons.py
+++ b/vantage6-common/vantage6/common/docker/addons.py
@@ -173,7 +173,7 @@ def inspect_remote_image_timestamp(docker_client, image: str, log=ClickLogger):
     if v1:
         timestamp = parse(result.json().get("created"))
     else:
-        timestamp = parse(result.json().get("extra_attrs").get("created"))
+        timestamp = parse(result.json().get("push_time"))
     return timestamp
 
 
diff --git a/vantage6/vantage6/cli/node.py b/vantage6/vantage6/cli/node.py
index 58d7f1ac..c58df6a9 100644
--- a/vantage6/vantage6/cli/node.py
+++ b/vantage6/vantage6/cli/node.py
@@ -354,8 +354,9 @@ def cli_node_start(name: str, config: str, environment: str,
         # docker_client.images.pull(image)
         pull_if_newer(docker.from_env(), image)
 
-    except Exception:
-        warning(' ... alas, no dice!')
+    except Exception as e:
+        warning(' ... Getting latest node image failed:')
+        warning(f"     {e}")
     else:
         info(" ... success!")
 
diff --git a/vantage6/vantage6/cli/server.py b/vantage6/vantage6/cli/server.py
index f1d43884..205939ef 100644
--- a/vantage6/vantage6/cli/server.py
+++ b/vantage6/vantage6/cli/server.py
@@ -192,8 +192,9 @@ def cli_server_start(ctx: Type[ServerContext], ip: str, port: int, image: str,
     try:
         pull_if_newer(docker.from_env(), image)
         # docker_client.images.pull(image)
-    except Exception:
-        warning("... alas, no dice!")
+    except Exception as e:
+        warning(' ... Getting latest node image failed:')
+        warning(f"     {e}")
     else:
         info(" ... success!")
 
@@ -493,8 +494,9 @@ def cli_server_import(ctx: Type[ServerContext], file_: str, drop_all: bool,
     info(f"Pulling latest server image '{image}'.")
     try:
         docker_client.images.pull(image)
-    except Exception:
-        warning("... alas, no dice!")
+    except Exception as e:
+        warning(' ... Getting latest node image failed:')
+        warning(f"     {e}")
     else:
         info(" ... success!")
Repository: vantage6/vantage6
Issue: [Bug] Pulling image is now based on `push_time` which is not updated if image is identical

**Describe the bug**
In `pull_if_newer()`, we pull the image if the remote timestamp is newer than the local one. Recently, we changed this (https://github.com/vantage6/vantage6/pull/482), and based the timestamp on the push time. 

It can occur that the local and remote image are identical, but that the remote push timestamp is newer. Then, the 'newer' remote image is pulled over and over again without ever updating the timestamp. This should be achieved somehow so that no pulls are attempted.

Note that when the images are identical, the pull time is short, so this issue is not very pressing for that reason.


Changed files:
vantage6-common/vantage6/common/docker/addons.py

diff --git a/vantage6-common/vantage6/common/docker/addons.py b/vantage6-common/vantage6/common/docker/addons.py
index e1ba4ae4..4a0c670b 100644
--- a/vantage6-common/vantage6/common/docker/addons.py
+++ b/vantage6-common/vantage6/common/docker/addons.py
@@ -149,7 +149,7 @@ def inspect_remote_image_timestamp(
         log.warn("Or an image from docker hub?")
         log.warn("We'll make a final attempt when running the image to pull"
                  " it without any checks...")
-        return
+        return None, None
 
     # figure out API of the docker repo
     v1_check = requests.get(f"https://{reg}/api/health")
@@ -163,7 +163,7 @@ def inspect_remote_image_timestamp(
         log.error(f"Could not determine version of the registry! {reg}")
         log.error("Is this a Harbor registry?")
         log.error("Or is the harbor server offline?")
-        return
+        return None, None
 
     if v1:
         image = f"https://{reg}/api/repositories/{rep}/{img_}/tags/{tag}"
@@ -179,12 +179,12 @@ def inspect_remote_image_timestamp(
     # verify that we got an result
     if result.status_code == 404:
         log.warn(f"Remote image not found! {image}")
-        return
+        return None, None
 
     if result.status_code != 200:
         log.warn(f"Remote info could not be fetched! ({result.status_code}) "
                  f"{image}")
-        return
+        return None, None
 
     if v1:
         timestamp = parse(result.json().get("created"))
@@ -221,10 +221,10 @@ def inspect_local_image_timestamp(
         img = docker_client.images.get(image)
     except docker.errors.ImageNotFound:
         log.debug(f"Local image does not exist! {image}")
-        return None
+        return None, None
     except docker.errors.APIError:
         log.debug(f"Local info not available! {image}")
-        return None
+        return None, None
 
     try:
         digest = img.attrs.get("RepoDigests")[0].split("@")[1]
Repository: vantage6/vantage6
Issue: [Bug] Users can submit socket events that are not intended for them

**Describe the bug**
There are two socket events (`on_node_info_update` and  `on_algorithm_status_change`) that are intended for nodes but which a user connected to a socket event could also access. They can't really use these to do anything harmful, but it should not be possible nevertheless.


Changed files:
vantage6-server/vantage6/server/websockets.py

diff --git a/vantage6-server/vantage6/server/websockets.py b/vantage6-server/vantage6/server/websockets.py
index 2b2328aa..4ef38785 100644
--- a/vantage6-server/vantage6/server/websockets.py
+++ b/vantage6-server/vantage6/server/websockets.py
@@ -227,6 +227,12 @@ class DefaultSocketNamespace(Namespace):
                 "collaboration_id": 1
             }
         """
+        # only allow nodes to send this event
+        if session.type != 'node':
+            self.log.warn('Only nodes can send algorithm status changes! '
+                          f'{session.type} {session.auth_id} is not allowed.')
+            return
+
         result_id = data.get('result_id')
         task_id = data.get('task_id')
         collaboration_id = data.get('collaboration_id')
@@ -272,6 +278,12 @@ class DefaultSocketNamespace(Namespace):
         node_config: dict
             Dictionary containing the node's configuration.
         """
+        # only allow nodes to send this event
+        if session.type != 'node':
+            self.log.warn('Only nodes can send node configuration updates! '
+                          f'{session.type} {session.auth_id} is not allowed.')
+            return
+
         node = db.Node.get(session.auth_id)
 
         # delete any old data that may be present (if cleanup on disconnect
Repository: spinnakermanchester/spinnfrontendcommon
Issue: This should be a FormatAdapter

On line:
https://github.com/SpiNNakerManchester/SpiNNFrontEndCommon/blob/1c9837468f678e27ec673170831b0899a0f2482d/spinn_front_end_common/interface/interface_functions/graph_data_specification_writer.py#L30

This needs to be a FormatAdapter to ensure the log later works.

Changed files:
spinn_front_end_common/utility_models/reverse_ip_tag_multi_cast_source.py
spinn_front_end_common/interface/interface_functions/graph_data_specification_writer.py

diff --git a/spinn_front_end_common/interface/interface_functions/graph_data_specification_writer.py b/spinn_front_end_common/interface/interface_functions/graph_data_specification_writer.py
index 155a84b23..e05136795 100644
--- a/spinn_front_end_common/interface/interface_functions/graph_data_specification_writer.py
+++ b/spinn_front_end_common/interface/interface_functions/graph_data_specification_writer.py
@@ -18,6 +18,7 @@ import logging
 
 from data_specification.constants import APP_PTR_TABLE_BYTE_SIZE
 from spinn_utilities.progress_bar import ProgressBar
+from spinn_utilities.log import FormatAdapter
 from data_specification import DataSpecificationGenerator
 from spinn_front_end_common.abstract_models import (
     AbstractRewritesDataSpecification, AbstractGeneratesDataSpecification)
@@ -27,7 +28,7 @@ from pacman.model.resources import MultiRegionSDRAM, ConstantSDRAM
 from data_specification.reference_context import ReferenceContext
 from spinn_front_end_common.utilities.utility_calls import get_report_writer
 
-logger = logging.getLogger(__name__)
+logger = FormatAdapter(logging.getLogger(__name__))
 
 
 def graph_data_specification_writer(
diff --git a/spinn_front_end_common/utility_models/reverse_ip_tag_multi_cast_source.py b/spinn_front_end_common/utility_models/reverse_ip_tag_multi_cast_source.py
index ed84c54da..769e75215 100644
--- a/spinn_front_end_common/utility_models/reverse_ip_tag_multi_cast_source.py
+++ b/spinn_front_end_common/utility_models/reverse_ip_tag_multi_cast_source.py
@@ -216,10 +216,10 @@ class ReverseIpTagMultiCastSource(
     def send_buffer_times(self, send_buffer_times):
         self._send_buffer_times = send_buffer_times
         for vertex in self.machine_vertices:
-            vertex_slice = vertex.vertex_slice
             send_buffer_times_to_set = self._send_buffer_times
-            if self._send_buffer_times:
+            if len(self._send_buffer_times) > 0:
                 if hasattr(self._send_buffer_times[0], "__len__"):
+                    vertex_slice = vertex.vertex_slice
                     send_buffer_times_to_set = self._send_buffer_times[
                         vertex_slice.lo_atom:vertex_slice.hi_atom + 1]
             vertex.send_buffer_times = send_buffer_times_to_set
Repository: spinnakermanchester/spinnfrontendcommon
Issue: chip_iobuf_clearer fails on second empty run

Script:
import spynnaker8 as sim
sim.setup(timestep=1.0)
sim.run(2)
sim.run(2)
sim.end()

Error:
  File ".../SpiNNFrontEndCommon/spinn_front_end_common/interface/interface_functions/chip_iobuf_clearer.py", line 30, in __call__
    executable_types[ExecutableType.USES_SIMULATION_INTERFACE]
KeyError: <ExecutableType.USES_SIMULATION_INTERFACE: 2>
Error in sys.excepthook:

Changed files:
spinn_front_end_common/interface/abstract_spinnaker_base.py

diff --git a/spinn_front_end_common/interface/abstract_spinnaker_base.py b/spinn_front_end_common/interface/abstract_spinnaker_base.py
index 87290e8d7..b09fb8b36 100644
--- a/spinn_front_end_common/interface/abstract_spinnaker_base.py
+++ b/spinn_front_end_common/interface/abstract_spinnaker_base.py
@@ -145,6 +145,9 @@ class AbstractSpinnakerBase(ConfigHandler, SimulatorInterface):
         # represent cores.
         "_original_machine_graph",
 
+        # boolean for empty graphs
+        "_empty_graphs",
+
         # the mapping interface between application and machine graphs.
         "_graph_mapper",
 
@@ -420,6 +423,7 @@ class AbstractSpinnakerBase(ConfigHandler, SimulatorInterface):
         self._original_application_graph = \\
             ApplicationGraph(label=self._graph_label)
         self._original_machine_graph = MachineGraph(label=self._graph_label)
+        self._empty_graphs = False
 
         self._graph_mapper = None
         self._placements = None
@@ -1502,6 +1506,7 @@ class AbstractSpinnakerBase(ConfigHandler, SimulatorInterface):
             if self._graph_mapper is not None:
                 inputs["MemoryGraphMapper"] = self._graph_mapper
         else:
+            self._empty_graphs = True
             logger.warning(
                 "Your graph has no vertices in it.")
             inputs["MemoryApplicationGraph"] = self._application_graph
@@ -1943,9 +1948,9 @@ class AbstractSpinnakerBase(ConfigHandler, SimulatorInterface):
                 "Reports", "write_sdram_usage_report_per_chip"):
             algorithms.append("SdramUsageReportPerChip")
 
-        # clear iobuf if were in multirun mode
+        # clear iobuf if we are in multirun mode
         if (self._has_ran and not graph_changed and
-                not self._use_virtual_board and
+                not self._use_virtual_board and not self._empty_graphs and
                 self._config.getboolean("Reports", "clear_iobuf_during_run")):
             algorithms.append("ChipIOBufClearer")
Repository: spinnakermanchester/spinnfrontendcommon
Issue: Tokens reused on second run no longer correct

class AbstractSpinnakerBase
method _do_load
block if not application_graph_changed and self._has_ran:
lines 1713 ...

tokens have changed but not here!

Also need to change MappingAlgorithms.md

Changed files:
spinn_front_end_common/interface/abstract_spinnaker_base.py

diff --git a/spinn_front_end_common/interface/abstract_spinnaker_base.py b/spinn_front_end_common/interface/abstract_spinnaker_base.py
index 2a5c3d407..31fd6e6cd 100644
--- a/spinn_front_end_common/interface/abstract_spinnaker_base.py
+++ b/spinn_front_end_common/interface/abstract_spinnaker_base.py
@@ -1712,14 +1712,6 @@ class AbstractSpinnakerBase(SimulatorInterface):
         if not application_graph_changed and self._has_ran:
             inputs["ExecutableTargets"] = self._last_run_outputs[
                 "ExecutableTargets"]
-            inputs["LoadedReverseIPTagsToken"] = self._last_run_outputs[
-                "LoadedReverseIPTagsToken"]
-            inputs["LoadedIPTagsToken"] = self._last_run_outputs[
-                "LoadedIPTagsToken"]
-            inputs["LoadedIPTagsToken"] = self._last_run_outputs[
-                "LoadedIPTagsToken"]
-            inputs["LoadedIPTagsToken"] = self._last_run_outputs[
-                "LoadedIPTagsToken"]
 
         algorithms = list()
Repository: spinnakermanchester/spinnfrontendcommon
Issue: recording utilities not supporting infinite runs of multiple populations

Discovered the following error message when connecting two populations (one with a live output). The addition of the non-live output population seemed to generate the error:

```
Generating sPyNNaker data specifications
|0                           50%                         100%|
 Exception "TypeError"
unsupported operand type(s) for *: 'NoneType' and 'int'
File: /home/rjames/SpiNNaker_devel/SpiNNFrontEndCommon/spinn_front_end_common/interface/buffer_management/recording_utilities.py, Line: 194
```

adding the following 'Alan Hack' to recording_utilities.py has prevented the error from occurring:

```python
# The size of each buffer is the actual size needed for the number of
    # timesteps, or the maximum for buffering if a maximum is specified
    if n_machine_time_steps is None:
        data = list()
        for n in buffered_sdram_per_timestep:
            data.append(0)
        return data
    else:
        return [
            n_machine_time_steps * sdram
            if (maximum_sdram_for_buffering is None or
                maximum_sdram_for_buffering[i] == 0 or
                (n_machine_time_steps * sdram) > maximum_sdram_for_buffering[i])
            else maximum_sdram_for_buffering
            for i, sdram in enumerate(buffered_sdram_per_timestep)
        ]
```


Changed files:
spinn_front_end_common/interface/buffer_management/recording_utilities.py

diff --git a/spinn_front_end_common/interface/buffer_management/recording_utilities.py b/spinn_front_end_common/interface/buffer_management/recording_utilities.py
index 880114fd6..7fbfb84d5 100644
--- a/spinn_front_end_common/interface/buffer_management/recording_utilities.py
+++ b/spinn_front_end_common/interface/buffer_management/recording_utilities.py
@@ -17,7 +17,7 @@ _LAST_SEQUENCE_NUMBER_OFFSET = 4 * 6
 _FIRST_REGION_ADDRESS_OFFSET = 4 * 7
 
 # the number of data elements inside the recording region before
-# recording regions iszes are stored.
+# recording regions sizes are stored.
 _RECORDING_ELEMENTS_BEFORE_REGION_SIZES = 7
 
 # The Buffer traffic type
@@ -189,6 +189,9 @@ def get_recorded_region_sizes(
 
     # The size of each buffer is the actual size needed for the number of
     # timesteps, or the maximum for buffering if a maximum is specified
+    if n_machine_time_steps is None:
+        data = [0 for _ in buffered_sdram_per_timestep]
+        return data
     return [
         n_machine_time_steps * sdram
         if (maximum_sdram_for_buffering is None or
Repository: spinnakermanchester/spinnfrontendcommon
Issue: _run_until_complete never turned off

if a user calls ASB run_until_complete and then ASB normal run the system will get very confused.
asb._run_until_complete is set to True

then _run_until_complete is used for many decisions
It is never turned off

Not sure it will ever happen but I did not find any safety code in place if it does.

Changed files:
spinn_front_end_common/interface/abstract_spinnaker_base.py

diff --git a/spinn_front_end_common/interface/abstract_spinnaker_base.py b/spinn_front_end_common/interface/abstract_spinnaker_base.py
index 7f591db90..a902fb289 100644
--- a/spinn_front_end_common/interface/abstract_spinnaker_base.py
+++ b/spinn_front_end_common/interface/abstract_spinnaker_base.py
@@ -801,6 +801,7 @@ class AbstractSpinnakerBase(ConfigHandler):
         """
         self._run_until_complete = True
         self._run(n_steps, sync_time=0)
+        self._run_until_complete = False
 
     def run(self, run_time, sync_time=0):
         """ Run a simulation for a fixed amount of time
Repository: spinnakermanchester/spinnfrontendcommon
Issue: The time scale factor could be reduced to 0.0

Running simple the LocalTDMABuilder claims
The time scale factor could be reduced to 0.0

Changed files:
spinn_front_end_common/interface/interface_functions/local_tdma_builder.py

diff --git a/spinn_front_end_common/interface/interface_functions/local_tdma_builder.py b/spinn_front_end_common/interface/interface_functions/local_tdma_builder.py
index afc6ed15f..a5dc23851 100644
--- a/spinn_front_end_common/interface/interface_functions/local_tdma_builder.py
+++ b/spinn_front_end_common/interface/interface_functions/local_tdma_builder.py
@@ -141,10 +141,6 @@ def local_tdma_builder(machine_graph, n_keys_map, application_graph=None):
         logger.warning(
             "A time scale factor of {} may be needed to run correctly"
             .format(time_scale_factor_needed))
-    elif max_fraction_of_sending < 1:
-        logger.info(
-            "The time scale factor could be reduced to {}"
-            .format(time_scale_factor_needed))
 
     # get initial offset for each app vertex.
     for app_vertex in application_graph.vertices:
Repository: spinnakermanchester/spinnfrontendcommon
Issue: Possible breakage in SystemMulticastRoutingGenerator

At the end of `SystemMulticastRoutingGenerator._add_routing_entries` we see this:

https://github.com/SpiNNakerManchester/SpiNNFrontEndCommon/blob/f5f1741a111b804a921af59b0537c1a471dc5787/spinn_front_end_common/interface/interface_functions/system_multicast_routing_generator.py#L180

That line is weird because it appears to be doing an increment that cannot have any effect. Should it be indented one level further so that it goes inside the loop immediately above it? That'd make sort of sense from a general-algorithmic-pattern sense.

Changed files:
spinn_front_end_common/interface/interface_functions/system_multicast_routing_generator.py

diff --git a/spinn_front_end_common/interface/interface_functions/system_multicast_routing_generator.py b/spinn_front_end_common/interface/interface_functions/system_multicast_routing_generator.py
index fa249ad2b..ea6fd7c87 100644
--- a/spinn_front_end_common/interface/interface_functions/system_multicast_routing_generator.py
+++ b/spinn_front_end_common/interface/interface_functions/system_multicast_routing_generator.py
@@ -221,4 +221,3 @@ class SystemMulticastRoutingGenerator(object):
                 link_ids=links_per_chip[x, y])
             # update tracker
             self._time_out_keys_by_board[(eth_x, eth_y)] = key
-        key += N_KEYS_PER_REINJECTION_PARTITION