Repository: datajoint/datajoint-python (language: python)

Issue:
Fix bug with unions.
Added handling for when union gets a union object inside its make_sql(), this allows you to do multiple unions.
Took the test from @guzman-raphael comments on the issue linked.

fix #926 
Python 3.10 error. AttributeError: module 'collections' has no attribute 'MutableMapping'
## Bug Report

### Description

`MutableMapping` has moved to: <https://docs.python.org/3/library/collections.abc.html#collections.abc.MutableMapping>

### Reproducibility
Include:
- OS `MACOS`
- Python Version `3.10.2`
- DataJoint Version `0.13.2`
- Minimum number of steps to reliably reproduce the issue

```
Python 3.10.2 | packaged by conda-forge | (main, Jan 14 2022, 08:04:21) [Clang 11.1.0 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import datajoint as dj
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/joseph/opt/conda/envs/testenv/lib/python3.10/site-packages/datajoint/__init__.py", line 30, in <module>
    from .settings import config
  File "/Users/joseph/opt/conda/envs/testenv/lib/python3.10/site-packages/datajoint/settings.py", line 61, in <module>
    class Config(collections.MutableMapping):
AttributeError: module 'collections' has no attribute 'MutableMapping'
```


`.descendants()` failed with master-part tables where the descendants only contain the part-table
## Bug Report

### Description
`table.descendants()` failed for the case where there are master-part tables downstream, where only the part-table is a descendant, but not the master.

### Reproducibility
Include:
- OS (WIN)
- Python 3.7
- MySQL Version 5.7
- MySQL Deployment Strategy (local-native)
- DataJoint Version 0.13.2

#### Reproducible case

Given the following schema design

```python
import datajoint as dj
schema = dj.Schema('test')

@schema
class A(dj.Manual):
    definition = """
    a: int
    """


@schema
class B(dj.Manual):
    definition = """
    -> A
    b: int
    """


@schema
class Master(dj.Manual):
    definition = """
    table_master: int
    """

    class Part(dj.Part):
        definition = """
        -> master
        -> B
        """
```

calling `A.descendants()` will result in the error below:

```
Traceback (most recent call last):
  File "C:\Users\thinh\Documents\TN-Vathes\MAP\map-ephys\venv\lib\site-packages\IPython\core\interactiveshell.py", line 3343, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-3-a68a3ab7b68c>", line 1, in <module>
    TableA.descendants()
  File "C:\Users\thinh\Documents\TN-Vathes\MAP\map-ephys\venv\lib\site-packages\datajoint\table.py", line 181, in descendants
    for node in self.connection.dependencies.descendants(self.full_table_name) if not node.isdigit()]
  File "C:\Users\thinh\Documents\TN-Vathes\MAP\map-ephys\venv\lib\site-packages\datajoint\dependencies.py", line 149, in descendants
    nx.algorithms.dag.topological_sort(nodes)))
  File "C:\Users\thinh\Documents\TN-Vathes\MAP\map-ephys\venv\lib\site-packages\datajoint\dependencies.py", line 30, in unite_master_parts
    raise DataJointError("Found a part table {name} without its master table.".format(name=name))
datajoint.errors.DataJointError: Found a part table `test`.`master_table__part_table` without its master table.
```
Cascading delete for Part
The cascading delete still doesn't handle the case involving `dj.Part` correctly. Basically, any deletion on `dj.Part` should cause deletion on its _parent_ table. Marking deletion on the _parent_ table then should cause appropriate cascading of deletion on the _parent's_ as well as `Part`'s subtables.

Drop all parts when master is dropped in a master-part relationship.
This is a rare problem but it can occur.

If a part table does not reference its master, then dropping the master will result in part tables with no master.  DataJoint does not know how to handle this situation and throws an error like this:
```
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
~/dev/datajoint-python/datajoint/schema.py in spawn_missing_classes(self)
    124             try:
--> 125                 master_class = self.context[to_camel_case(groups['master'])]
    126             except KeyError:

KeyError: 'Response'

During handling of the above exception, another exception occurred:

DataJointError                            Traceback (most recent call last)
<ipython-input-4-fffe84c8c428> in <module>()
      1 from pipeline import experiment, reso, meso, shared
----> 2 from stimline import stimulus, tune

~/dev/stimuli/python/stimline/tune.py in <module>()
    122 
    123 @schema
--> 124 class Response(dj.Computed):
    125     definition = """  # response to directional stimulus
    126     -> Activity

~/dev/stimuli/python/stimline/tune.py in Response()
    185 
    186 
--> 187     schema.spawn_missing_classes()

~/dev/datajoint-python/datajoint/schema.py in spawn_missing_classes(self)
    125                 master_class = self.context[to_camel_case(groups['master'])]
    126             except KeyError:
--> 127                 raise DataJointError('The table %s does not follow DataJoint naming conventions' % table_name)
    128             part_class = type(class_name, (Part,), dict(definition=...))
    129             part_class._master = master_class

DataJointError: The table __response__meso_trial does not follow DataJoint naming conventions
```

Although it's unusual to create part tables that don't reference their masters, it is possible and this needs to be fixed.

I propose the following fix:  Dropping a master table should result in dropping of all its parts regardless of whether they make a foreign key reference to the master or not.
Cannot form the union of more than two tables
## Bug Report

### Description
I have a situation where I need to take the union over multiple tables (restriction or other methods will not work in this case). Taking the union over more than two tables throws an error.

### Reproducibility
Include:
- OS (MACOS)
- Python Version 3.8.8
- MySQL Deployment Strategy (remote)
- DataJoint Version 0.13.2

This is a trivial example that seems to reproduce the issue:
```python
@schema
class Params(dj.Lookup):
    definition = """
    params_id: tinyint unsigned
    """

Params.insert([[x] for x in range(3)])

params = [Params & {'params_id': x} for x in range(3)]

params[0] + params[1] + params[2]
```

Error stack:
```console
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
~/anaconda3/envs/neural-data-kit/lib/python3.8/site-packages/IPython/core/formatters.py in __call__(self, obj)
    700                 type_pprinters=self.type_printers,
    701                 deferred_pprinters=self.deferred_printers)
--> 702             printer.pretty(obj)
    703             printer.flush()
    704             return stream.getvalue()

~/anaconda3/envs/neural-data-kit/lib/python3.8/site-packages/IPython/lib/pretty.py in pretty(self, obj)
    392                         if cls is not object \
    393                                 and callable(cls.__dict__.get('__repr__')):
--> 394                             return _repr_pprint(obj, self, cycle)
    395 
    396             return _default_pprint(obj, self, cycle)

~/anaconda3/envs/neural-data-kit/lib/python3.8/site-packages/IPython/lib/pretty.py in _repr_pprint(obj, p, cycle)
    698     """A pprint that just redirects to the normal repr function."""
    699     # Find newlines and replace them with p.break_()
--> 700     output = repr(obj)
    701     lines = output.splitlines()
    702     with p.group():

~/anaconda3/envs/neural-data-kit/lib/python3.8/site-packages/datajoint/expression.py in __repr__(self)
    527         :rtype: str
    528         """
--> 529         return super().__repr__() if config['loglevel'].lower() == 'debug' else self.preview()
    530 
    531     def preview(self, limit=None, width=None):

~/anaconda3/envs/neural-data-kit/lib/python3.8/site-packages/datajoint/expression.py in preview(self, limit, width)
    531     def preview(self, limit=None, width=None):
    532         """ :return: a string of preview of the contents of the query. """
--> 533         return preview(self, limit, width)
    534 
    535     def _repr_html_(self):

~/anaconda3/envs/neural-data-kit/lib/python3.8/site-packages/datajoint/preview.py in preview(query_expression, limit, width)
     11     if width is None:
     12         width = config['display.width']
---> 13     tuples = rel.fetch(limit=limit + 1, format="array")
     14     has_more = len(tuples) > limit
     15     tuples = tuples[:limit]

~/anaconda3/envs/neural-data-kit/lib/python3.8/site-packages/datajoint/fetch.py in __call__(self, offset, limit, order_by, format, as_dict, squeeze, download_path, *attrs)
    190                 ret = return_values[0] if len(attrs) == 1 else return_values
    191         else:  # fetch all attributes as a numpy.record_array or pandas.DataFrame
--> 192             cur = self._expression.cursor(
    193                 as_dict=as_dict, limit=limit, offset=offset, order_by=order_by)
    194             heading = self._expression.heading

~/anaconda3/envs/neural-data-kit/lib/python3.8/site-packages/datajoint/expression.py in cursor(self, offset, limit, order_by, as_dict)
    511         if offset and limit is None:
    512             raise DataJointError('limit is required when offset is set')
--> 513         sql = self.make_sql()
    514         if order_by is not None:
    515             sql += ' ORDER BY ' + ', '.join(order_by)

~/anaconda3/envs/neural-data-kit/lib/python3.8/site-packages/datajoint/expression.py in make_sql(self, fields)
    111             distinct="DISTINCT " if distinct else "",
    112             fields=self.heading.as_sql(fields or self.heading.names),
--> 113             from_=self.from_clause(), where=self.where_clause())
    114 
    115     # --------- query operators -----------

~/anaconda3/envs/neural-data-kit/lib/python3.8/site-packages/datajoint/expression.py in from_clause(self)
     91             self._subquery_alias_count) if isinstance(src, QueryExpression)
     92             else src for src in self.support)
---> 93         clause = next(support)
     94         for s, left in zip(support, self._left):
     95             clause += ' NATURAL{left} JOIN {clause}'.format(

~/anaconda3/envs/neural-data-kit/lib/python3.8/site-packages/datajoint/expression.py in (.0)
     88 
     89     def from_clause(self):
---> 90         support = ('(' + src.make_sql() + ') as `$%x`' % next(
     91             self._subquery_alias_count) if isinstance(src, QueryExpression)
     92             else src for src in self.support)

~/anaconda3/envs/neural-data-kit/lib/python3.8/site-packages/datajoint/expression.py in make_sql(self)
    627             fields = arg1.primary_key
    628             return "({sql1}) UNION ({sql2})".format(
--> 629                 sql1=arg1.make_sql(fields),
    630                 sql2=arg2.make_sql(fields))
    631         # with secondary attributes, use union of left join with antijoin

TypeError: make_sql() takes 1 positional argument but 2 were given
```

### Expected Behavior
Forming the union over two elements of `params` works as expected with the same setup as described above.

Note: taking the union over more than two tables **does** work for python version 3.8.2 and datajoint version 0.12.6.

Avoid or warn avoid dropping when relation has been restricted
Something like (Segmentation() & my_key).drop() should maybe warn you about whether you want to delete rather than drop. Right now, it does the same as Segmentation().drop()
`.descendants()` failed with master-part tables where the descendants only contain the part-table
## Bug Report

### Description
`table.descendants()` failed for the case where there are master-part tables downstream, where only the part-table is a descendant, but not the master.

### Reproducibility
Include:
- OS (WIN)
- Python 3.7
- MySQL Version 5.7
- MySQL Deployment Strategy (local-native)
- DataJoint Version 0.13.2

#### Reproducible case

Given the following schema design

```python
import datajoint as dj
schema = dj.Schema('test')

@schema
class A(dj.Manual):
    definition = """
    a: int
    """


@schema
class B(dj.Manual):
    definition = """
    -> A
    b: int
    """


@schema
class Master(dj.Manual):
    definition = """
    table_master: int
    """

    class Part(dj.Part):
        definition = """
        -> master
        -> B
        """
```

calling `A.descendants()` will result in the error below:

```
Traceback (most recent call last):
  File "C:\Users\thinh\Documents\TN-Vathes\MAP\map-ephys\venv\lib\site-packages\IPython\core\interactiveshell.py", line 3343, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-3-a68a3ab7b68c>", line 1, in <module>
    TableA.descendants()
  File "C:\Users\thinh\Documents\TN-Vathes\MAP\map-ephys\venv\lib\site-packages\datajoint\table.py", line 181, in descendants
    for node in self.connection.dependencies.descendants(self.full_table_name) if not node.isdigit()]
  File "C:\Users\thinh\Documents\TN-Vathes\MAP\map-ephys\venv\lib\site-packages\datajoint\dependencies.py", line 149, in descendants
    nx.algorithms.dag.topological_sort(nodes)))
  File "C:\Users\thinh\Documents\TN-Vathes\MAP\map-ephys\venv\lib\site-packages\datajoint\dependencies.py", line 30, in unite_master_parts
    raise DataJointError("Found a part table {name} without its master table.".format(name=name))
datajoint.errors.DataJointError: Found a part table `test`.`master_table__part_table` without its master table.
```


Discussion / hints:



The term "subtable", as we have sometimes used it, is now replaced with "part relation". Let's also use "master relation" to indicate the relation containing the part relation.  We have also decided that part relations cannot have their own part relations.

So to restate the present issue: 
- Deleting from a part relation should not be possible without deleting matching tuples from its master relation. 

We have two options to address this issue:
1. prohibit deletes from a part relation that are not cascaded through its master relation 
2. delete from the master relation when a delete is issued on the part relation

Option 1 seems overly restrictive since a part relation may have dependencies not shared by its master and some cascading deletes may halt and force the user to break up a single delete command into several more specific commands.  If we pursue this solution, we need to make sure that the error message and the suggested course of action are clear.

Option 2 seems a bit aggressive since now deletes will cascade one level up the dependency hierarchy as well as down. This may be counterintuitive to some users.  This concern is partly addressed by displaying a summary of the data to be deleted before committing the delete.  Also option 2 may produce more complex queries that may be much slower than ones resulting from Option 1.

Which option should we choose?

I almost always prefer strong and convenient (but potentially destructive) solution, provided that we keep it informative. So if I have to choose, I'd prefer to use option 2. However, I also think this is a perfect opportunity for us to use config to alter between different behaviors, and offer both option 1 and option 2.

I prefer option 2 and I think we should offer to change the behavior in the config. 

I support Option 2. However, making datajoint configurable between these two options would create undue documentation and testing burden. Let's implement Option 2 and document it well.

I relabeled this as a 'feature' because the current implementation acts reasonably: it fails to delete when attempting to delete from parts, bypassing the master. I just don't want panic to set in that cascading deletes are unsafe.

Hello! OK, so the term "subtable" is no longer used. It's crucial to update the documentation to reflect that. I just spent several hours trying to figure out how to define a subtable, including searching through the code, and finally ended up here.

Am I correct in assuming the original issue reported above by @eywalker remains? I'm trying to define a part relation, such that if I try to delete parts, I'm either not allowed to, or I am but it automatically deletes the appropriate tuples in the parent (or "master part"). At the moment, I'm not seeing either option 1 or 2 behaviour. I've followed the example I found in `datajoint.tests.schema.Ephys.Channel`. Maybe the easiest thing to do for now is to define a `delete()` method that does nothing? I've added this to my part relation:
```
        def delete(self):
            """Override self.delete() so that it does nothing other than print a message"""
            print("direct deletion from %s is disabled because it's a part relation "
                  "(subtable) of %r" % (self.__class__, self._master))
```
and it seems to work as intended.
Hi @mspacek, the master-part relationship has now been defined consistently between matlab and python and we can move quickly on the documentation.  Thank you for your patience.

Yes, this issue is still open.  The master-part relationship has not been strongly enforced.  Now you are able to delete from part without deleting from master.  This should not be done and eventually we need to enforce the master-part discipline by either Option 1 or Option 2.  I like Option 1 because it is easy to implement correctly.  Option 2 is more involved but may be more user friendly.
I propose to implement Option 1 now similar to what @mspacek did.   If we run into problems, we will create a new issue to implement Option 2.
Let's proceed with Option 1 for now, but plan to implement Option 2 in the future. There wouldn't be a problem per se with Option 1 but it is undoubtedly more convenient to have Option 2 available as a feature. Future transition from Option 1 -> Option 2 should cause no issue.
Support for this will significantly improve the usability of `Part` tables and any future extensions around this logic.
Hi guys,
judging from the earlier conversation I am not sure if you are aware of this: while direct (manual) deletes from a `Part` table fail, deletes that are caused by deleting from a referenced table will go through and delete only from `Part`, leaving `Master` orphaned. Contrary to what @dimitri-yatsenko wrote earlier I would thus conclude that cascading deletes are indeed unsafe. As I couldn't really figure out how to do properly construct the delete list from sql as it is done in the datajoint code, I tried to write a modified delete method that looks up the `Master` tables in python and performs the deletes:
```python
 def delete_cascade(self):
       self.connection.dependencies.load()
       master_tables_of_descendants = []
       keys_to_delete = []
       for desc in self.descendants():
           desc_class_name = dj.table.lookup_class_name(desc, globals())
           desc_class_instance = eval(desc_class_name+'()')
           try:
               master_tables_of_descendants.append(desc_class_instance._master)
               keys_to_delete.append((desc_class_instance._master & self).fetch(dj.key))
           except AttributeError:
               pass
        for master, keys in zip(master_tables_of_descendants, keys_to_delete):
            (master() & keys).delete()
        self.delete()
```
Perhaps something similar could already be done as a hotfix before the bug is properly fixed in a future release?
I just noticed that the same problem also exists for `drop`.
High priority for release 0.12.5
Possibly related effort with #374 
Yes, #374 and #151 can be merged and addressed with a single solution.
@dimitri-yatsenko should parts be allowed to not reference master? 

Also, if so, this issue can probably be closed based on review of merge history ^^.




Given that restriction makes no sense in `drop` and this is likely mistype for `delete`, I agree that it makes sense to throw warning or even an error.
Yes, let's throw an error. 

Should insert throw an error when invoked on a restricted set? 

```python
(Segmentation() & my_key).insert1(tup)
```

Shouldn't `populate` also?
I feel that the `drop` is the only case for which I feel strongly about not allowing operation on a restricted table - because it suggests that one has mistaken `drop` for `delete` but with devastatingly different outcomes.

 For other operations, it is rather convenient to be able to just work with restricted relation even if restriction would not mean anything. 


This goes back to the topic of class methods vs instance methods. Things like `drop` refer to the class rather than to the object.  Perhaps we could make it so drop could not be called on an instance.  

We could make it so `Scan().drop()` will throw the error "Drop should be called on the class"

The correct way would be 
```python
Scan.drop()
```
Given that it's a bit awkward to have a class method that'll behave differently if it's called on an instance (and since making that happen would require us to write our own `Descriptor` type - which feels like an overkill), I vote that we take the minimal action of disallowing `drop` called on a restricted table.


Fix patch:
diff --git a/.github/workflows/development.yaml b/.github/workflows/development.yaml
index c4c2b3475..39308c026 100644
--- a/.github/workflows/development.yaml
+++ b/.github/workflows/development.yaml
@@ -40,7 +40,7 @@ jobs:
           PY_VER: ${{matrix.py_ver}}
           MYSQL_VER: ${{matrix.mysql_ver}}
           ALPINE_VER: "3.10"
-          MINIO_VER: RELEASE.2019-09-26T19-42-35Z
+          MINIO_VER: RELEASE.2021-09-03T03-56-13Z
           COMPOSE_HTTP_TIMEOUT: "120"
           COVERALLS_SERVICE_NAME: travis-ci
           COVERALLS_REPO_TOKEN: fd0BoXG46TPReEem0uMy7BJO5j0w1MQiY
diff --git a/CHANGELOG.md b/CHANGELOG.md
index bce0d7ccf..f8ce167f0 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,5 +1,24 @@
 ## Release notes
 
+### 0.13.3 -- Feb 9, 2022
+* Bugfix - Fix error in listing ancestors, descendants with part tables.
+* Bugfix - Fix Python 3.10 compatibility (#983) PR #972
+* Bugfix - Allow renaming non-conforming attributes in proj (#982) PR #972
+* Add - Expose proxy feature for S3 external stores (#961) PR #962
+* Add - implement multiprocessing in populate (#695) PR #704, #969
+* Bugfix - Dependencies not properly loaded on populate. (#902) PR #919
+* Bugfix - Replace use of numpy aliases of built-in types with built-in type. (#938) PR #939
+* Bugfix - Deletes and drops must include the master of each part. (#151, #374) PR #957
+* Bugfix - `ExternalTable.delete` should not remove row on error (#953) PR #956
+* Bugfix - Fix error handling of remove_object function in `s3.py` (#952) PR #955
+* Bugfix - Fix regression issue with `DISTINCT` clause and `GROUP_BY` (#914) PR #963
+* Bugfix - Fix sql code generation to comply with sql mode `ONLY_FULL_GROUP_BY` (#916) PR #965
+* Bugfix - Fix count for left-joined `QueryExpressions` (#951) PR #966
+* Bugfix - Fix assertion error when performing a union into a join (#930) PR #967
+* Update `~jobs.error_stack` from blob to mediumblob to allow error stacks >64kB in jobs (#984) PR #986
+* Bugfix - Fix error when performing a union on multiple tables (#926) PR #964
+* Add - Allow optional keyword arguments for `make()` in `populate()` PR #971
+
 ### 0.13.2 -- May 7, 2021
 * Update `setuptools_certificate` dependency to new name `otumat`
 * Bugfix - Explicit calls to `dj.Connection` throw error due to missing `host_input` (#895) PR #907
@@ -41,13 +60,13 @@
 * Fix display of part tables in `schema.save`. (#821) PR #833
 * Add `schema.list_tables`. (#838) PR #844
 * Fix minio new version regression.  PR #847
-* Add more S3 logging for debugging. (#831) PR #832 
+* Add more S3 logging for debugging. (#831) PR #832
 * Convert testing framework from TravisCI to GitHub Actions (#841) PR #840
- 
+
 ### 0.12.7 -- Oct 27, 2020
 * Fix case sensitivity issues to adapt to MySQL 8+.  PR #819
 * Fix pymysql regression bug (#814) PR #816
-* Adapted attribute types now have dtype=object in all recarray results. PR #811 
+* Adapted attribute types now have dtype=object in all recarray results. PR #811
 
 ### 0.12.6 -- May 15, 2020
 * Add `order_by` to `dj.kill` (#668, #779) PR #775, #783
@@ -119,7 +138,7 @@
 * Fix #628 - incompatibility with pyparsing 2.4.1
 
 ### 0.11.1 -- Nov 15, 2018
-* Fix ordering of attributes in proj (#483 and #516)
+* Fix ordering of attributes in proj (#483, #516)
 * Prohibit direct insert into auto-populated tables (#511)
 
 ### 0.11.0 -- Oct 25, 2018
@@ -139,9 +158,9 @@
 * Bugfix in restriction of the form (A & B) * B (#463)
 * Improved error messages (#466)
 
-### 0.10.0 -- Jan 10, 2018 
+### 0.10.0 -- Jan 10, 2018
 * Deletes are more efficient (#424)
-* ERD shows table definition on tooltip hover in Jupyter (#422) 
+* ERD shows table definition on tooltip hover in Jupyter (#422)
 * S3 external storage
 * Garbage collection for external sorage
 * Most operators and methods of tables can be invoked as class methods rather than instance methods (#407)
@@ -155,7 +174,7 @@
 * Implement union operator +
 * Implement file-based external storage
 
-### 0.8.0 -- Jul 26, 2017 
+### 0.8.0 -- Jul 26, 2017
 Documentation and tutorials available at https://docs.datajoint.io and https://tutorials.datajoint.io
 * improved the ERD graphics and features using the graphviz libraries (#207, #333)
 * improved password handling logic (#322, #321)
@@ -174,11 +193,11 @@ Documentation and tutorials available at https://docs.datajoint.io and https://t
 * Added `dj.create_virtual_module`
 
 ### 0.4.10 (#286) -- Feb 6, 2017
-* Removed Vagrant and Readthedocs support 
+* Removed Vagrant and Readthedocs support
 * Explicit saving of configuration (issue #284)
 
 ### 0.4.9 (#285) -- Feb 2, 2017
-* Fixed setup.py for pip install 
+* Fixed setup.py for pip install
 
 ### 0.4.7 (#281) -- Jan 24, 2017
 * Fixed issues related to order of attributes in projection.
@@ -207,10 +226,10 @@ Documentation and tutorials available at https://docs.datajoint.io and https://t
 
 ### 0.3.8  -- Aug 2, 2016
 * added the `_update` method in `base_relation`. It allows updating values in existing tuples.
-* bugfix in reading values of type double.  Previously it was cast as float32. 
+* bugfix in reading values of type double.  Previously it was cast as float32.
 
 ### 0.3.7  -- Jul 31, 2016
-* added parameter `ignore_extra_fields` in `insert` 
+* added parameter `ignore_extra_fields` in `insert`
 * `insert(..., skip_duplicates=True)` now relies on `SELECT IGNORE`.  Previously it explicitly checked if tuple already exists.
 * table previews now include blob attributes displaying the string <BLOB>
 
@@ -232,7 +251,7 @@ Documentation and tutorials available at https://docs.datajoint.io and https://t
 * ERD() no longer text the context argument.
 * ERD.draw() now takes an optional context argument.  By default uses the caller's locals.
 
-### 0.3.2.   
+### 0.3.2.
 * Fixed issue #223:  `insert` can insert relations without fetching.
 * ERD() now takes the `context` argument, which specifies in which context to look for classes. The default is taken from the argument (schema or relation).
 * ERD.draw() no longer has the `prefix` argument: class names are shown as found in the context.
diff --git a/LNX-docker-compose.yml b/LNX-docker-compose.yml
index 448ab6f02..c6367c3c1 100644
--- a/LNX-docker-compose.yml
+++ b/LNX-docker-compose.yml
@@ -32,7 +32,7 @@ services:
       interval: 1s
   fakeservices.datajoint.io:
     <<: *net
-    image: datajoint/nginx:v0.0.16
+    image: datajoint/nginx:v0.0.19
     environment:
     - ADD_db_TYPE=DATABASE
     - ADD_db_ENDPOINT=db:3306
diff --git a/README.md b/README.md
index a5ee24e30..c08f59599 100644
--- a/README.md
+++ b/README.md
@@ -108,7 +108,7 @@ A number of labs are currently adopting DataJoint and we are quickly getting the
 PY_VER=3.7
 ALPINE_VER=3.10
 MYSQL_VER=5.7
-MINIO_VER=RELEASE.2019-09-26T19-42-35Z
+MINIO_VER=RELEASE.2021-09-03T03-56-13Z
 UID=1000
 GID=1000
 ```
@@ -136,6 +136,8 @@ GID=1000
 * Add entry in `/etc/hosts` for `127.0.0.1 fakeservices.datajoint.io`
 
 
+
+
 ### Launch Jupyter Notebook for Interactive Use
 * Navigate to `localhost:8888`
 * Input Jupyter password
diff --git a/datajoint/autopopulate.py b/datajoint/autopopulate.py
index 9e9615144..b60441599 100644
--- a/datajoint/autopopulate.py
+++ b/datajoint/autopopulate.py
@@ -8,12 +8,37 @@
 from .expression import QueryExpression, AndList
 from .errors import DataJointError, LostConnectionError
 import signal
+import multiprocessing as mp
 
 # noinspection PyExceptionInherit,PyCallingNonCallable
 
 logger = logging.getLogger(__name__)
 
 
+# --- helper functions for multiprocessing --
+
+def _initialize_populate(table, jobs, populate_kwargs):
+    """
+    Initialize the process for mulitprocessing.
+    Saves the unpickled copy of the table to the current process and reconnects.
+    """
+    process = mp.current_process()
+    process.table = table
+    process.jobs = jobs
+    process.populate_kwargs = populate_kwargs
+    table.connection.connect()  # reconnect
+
+
+def _call_populate1(key):
+    """
+    Call current process' table._populate1()
+    :key - a dict specifying job to compute
+    :return: key, error if error, otherwise None
+    """
+    process = mp.current_process()
+    return process.table._populate1(key, process.jobs, **process.populate_kwargs)
+
+
 class AutoPopulate:
     """
     AutoPopulate is a mixin class that adds the method populate() to a Relation class.
@@ -26,21 +51,23 @@ class AutoPopulate:
     @property
     def key_source(self):
         """
-        :return: the relation whose primary key values are passed, sequentially, to the
-                ``make`` method when populate() is called.
-                The default value is the join of the parent relations.
-                Users may override to change the granularity or the scope of populate() calls.
+        :return: the query expression that yields primary key values to be passed,
+        sequentially, to the ``make`` method when populate() is called.
+        The default value is the join of the parent tables references from the primary key.
+        Subclasses may override they key_source to change the scope or the granularity
+        of the make calls.
         """
         def _rename_attributes(table, props):
             return (table.proj(
                 **{attr: ref for attr, ref in props['attr_map'].items() if attr != ref})
-                if props['aliased'] else table)
+                if props['aliased'] else table.proj())
 
         if self._key_source is None:
-            parents = self.target.parents(primary=True, as_objects=True, foreign_key_info=True)
+            parents = self.target.parents(
+                primary=True, as_objects=True, foreign_key_info=True)
             if not parents:
-                raise DataJointError(
-                    'A relation must have primary dependencies for auto-populate to work')
+                raise DataJointError('A table must have dependencies '
+                                     'from its primary key for auto-populate to work')
             self._key_source = _rename_attributes(*parents[0])
             for q in parents[1:]:
                 self._key_source *= _rename_attributes(*q)
@@ -48,17 +75,19 @@ def _rename_attributes(table, props):
 
     def make(self, key):
         """
-        Derived classes must implement method `make` that fetches data from tables that are
-        above them in the dependency hierarchy, restricting by the given key, computes dependent
-        attributes, and inserts the new tuples into self.
+        Derived classes must implement method `make` that fetches data from tables
+        above them in the dependency hierarchy, restricting by the given key,
+        computes secondary attributes, and inserts the new tuples into self.
         """
-        raise NotImplementedError('Subclasses of AutoPopulate must implement the method `make`')
+        raise NotImplementedError(
+            'Subclasses of AutoPopulate must implement the method `make`')
 
     @property
     def target(self):
         """
         :return: table to be populated.
-        In the typical case, dj.AutoPopulate is mixed into a dj.Table class by inheritance and the target is self.
+        In the typical case, dj.AutoPopulate is mixed into a dj.Table class by
+        inheritance and the target is self.
         """
         return self
 
@@ -85,29 +114,40 @@ def _jobs_to_do(self, restrictions):
 
         if not isinstance(todo, QueryExpression):
             raise DataJointError('Invalid key_source value')
-        # check if target lacks any attributes from the primary key of key_source
+
         try:
+            # check if target lacks any attributes from the primary key of key_source
             raise DataJointError(
-                'The populate target lacks attribute %s from the primary key of key_source' % next(
-                    name for name in todo.heading.primary_key if name not in self.target.heading))
+                'The populate target lacks attribute %s '
+                'from the primary key of key_source' % next(
+                    name for name in todo.heading.primary_key
+                    if name not in self.target.heading))
         except StopIteration:
             pass
         return (todo & AndList(restrictions)).proj()
 
     def populate(self, *restrictions, suppress_errors=False, return_exception_objects=False,
                  reserve_jobs=False, order="original", limit=None, max_calls=None,
-                 display_progress=False):
+                 display_progress=False, processes=1, make_kwargs=None):
         """
-        rel.populate() calls rel.make(key) for every primary key in self.key_source
-        for which there is not already a tuple in rel.
-        :param restrictions: a list of restrictions each restrict (rel.key_source - target.proj())
+        ``table.populate()`` calls ``table.make(key)`` for every primary key in
+        ``self.key_source`` for which there is not already a tuple in table.
+
+        :param restrictions: a list of restrictions each restrict
+            (table.key_source - target.proj())
         :param suppress_errors: if True, do not terminate execution.
         :param return_exception_objects: return error objects instead of just error messages
-        :param reserve_jobs: if true, reserves job to populate in asynchronous fashion
+        :param reserve_jobs: if True, reserve jobs to populate in asynchronous fashion
         :param order: "original"|"reverse"|"random"  - the order of execution
+        :param limit: if not None, check at most this many keys
+        :param max_calls: if not None, populate at most this many keys
         :param display_progress: if True, report progress_bar
-        :param limit: if not None, checks at most that many keys
-        :param max_calls: if not None, populates at max that many keys
+        :param processes: number of processes to use. When set to a large number, then
+            uses as many as CPU cores
+        :param make_kwargs: Keyword arguments which do not affect the result of computation
+            to be passed down to each ``make()`` call. Computation arguments should be
+            specified within the pipeline e.g. using a `dj.Lookup` table.
+        :type make_kwargs: dict, optional
         """
         if self.connection.in_transaction:
             raise DataJointError('Populate cannot be called during a transaction.')
@@ -115,10 +155,9 @@ def populate(self, *restrictions, suppress_errors=False, return_exception_object
         valid_order = ['original', 'reverse', 'random']
         if order not in valid_order:
             raise DataJointError('The order argument must be one of %s' % str(valid_order))
-        error_list = [] if suppress_errors else None
         jobs = self.connection.schemas[self.target.database].jobs if reserve_jobs else None
 
-        # define and setup signal handler for SIGTERM
+        # define and set up signal handler for SIGTERM:
         if reserve_jobs:
             def handler(signum, frame):
                 logger.info('Populate terminated by SIGTERM')
@@ -131,60 +170,100 @@ def handler(signum, frame):
         elif order == "random":
             random.shuffle(keys)
 
-        call_count = 0
         logger.info('Found %d keys to populate' % len(keys))
 
-        make = self._make_tuples if hasattr(self, '_make_tuples') else self.make
+        keys = keys[:max_calls]
+        nkeys = len(keys)
 
-        for key in (tqdm(keys, desc=self.__class__.__name__) if display_progress else keys):
-            if max_calls is not None and call_count >= max_calls:
-                break
-            if not reserve_jobs or jobs.reserve(self.target.table_name, self._job_key(key)):
-                self.connection.start_transaction()
-                if key in self.target:  # already populated
-                    self.connection.cancel_transaction()
-                    if reserve_jobs:
-                        jobs.complete(self.target.table_name, self._job_key(key))
+        if processes > 1:
+            processes = min(processes, nkeys, mp.cpu_count())
+
+        error_list = []
+        populate_kwargs = dict(
+            suppress_errors=suppress_errors,
+            return_exception_objects=return_exception_objects,
+            make_kwargs=make_kwargs)
+
+        if processes == 1:
+            for key in tqdm(keys, desc=self.__class__.__name__) if display_progress else keys:
+                error = self._populate1(key, jobs, **populate_kwargs)
+                if error is not None:
+                    error_list.append(error)
+        else:
+            # spawn multiple processes
+            self.connection.close()  # disconnect parent process from MySQL server
+            del self.connection._conn.ctx  # SSLContext is not pickleable
+            with mp.Pool(processes, _initialize_populate, (self, populate_kwargs)) as pool:
+                if display_progress:
+                    with tqdm(desc="Processes: ", total=nkeys) as pbar:
+                        for error in pool.imap(_call_populate1, keys, chunksize=1):
+                            if error is not None:
+                                error_list.append(error)
+                            pbar.update()
                 else:
-                    logger.info('Populating: ' + str(key))
-                    call_count += 1
-                    self.__class__._allow_insert = True
-                    try:
-                        make(dict(key))
-                    except (KeyboardInterrupt, SystemExit, Exception) as error:
-                        try:
-                            self.connection.cancel_transaction()
-                        except LostConnectionError:
-                            pass
-                        error_message = '{exception}{msg}'.format(
-                            exception=error.__class__.__name__,
-                            msg=': ' + str(error) if str(error) else '')
-                        if reserve_jobs:
-                            # show error name and error message (if any)
-                            jobs.error(
-                                self.target.table_name, self._job_key(key),
-                                error_message=error_message, error_stack=traceback.format_exc())
-                        if not suppress_errors or isinstance(error, SystemExit):
-                            raise
-                        else:
-                            logger.error(error)
-                            error_list.append((key, error if return_exception_objects else error_message))
-                    else:
-                        self.connection.commit_transaction()
-                        if reserve_jobs:
-                            jobs.complete(self.target.table_name, self._job_key(key))
-                    finally:
-                        self.__class__._allow_insert = False
+                    for error in pool.imap(_call_populate1, keys):
+                        if error is not None:
+                            error_list.append(error)
+            self.connection.connect()  # reconnect parent process to MySQL server
 
-        # place back the original signal handler
+        # restore original signal handler:
         if reserve_jobs:
             signal.signal(signal.SIGTERM, old_handler)
-        return error_list
+
+        if suppress_errors:
+            return error_list
+
+    def _populate1(self, key, jobs, suppress_errors, return_exception_objects, make_kwargs=None):
+        """
+        populates table for one source key, calling self.make inside a transaction.
+        :param jobs: the jobs table or None if not reserve_jobs
+        :param key: dict specifying job to populate
+        :param suppress_errors: bool if errors should be suppressed and returned
+        :param return_exception_objects: if True, errors must be returned as objects
+        :return: (key, error) when suppress_errors=True, otherwise None
+        """
+        make = self._make_tuples if hasattr(self, '_make_tuples') else self.make
+
+        if jobs is None or jobs.reserve(self.target.table_name, self._job_key(key)):
+            self.connection.start_transaction()
+            if key in self.target:  # already populated
+                self.connection.cancel_transaction()
+                if jobs is not None:
+                    jobs.complete(self.target.table_name, self._job_key(key))
+            else:
+                logger.info('Populating: ' + str(key))
+                self.__class__._allow_insert = True
+                try:
+                    make(dict(key), **(make_kwargs or {}))
+                except (KeyboardInterrupt, SystemExit, Exception) as error:
+                    try:
+                        self.connection.cancel_transaction()
+                    except LostConnectionError:
+                        pass
+                    error_message = '{exception}{msg}'.format(
+                        exception=error.__class__.__name__,
+                        msg=': ' + str(error) if str(error) else '')
+                    if jobs is not None:
+                        # show error name and error message (if any)
+                        jobs.error(
+                            self.target.table_name, self._job_key(key),
+                            error_message=error_message, error_stack=traceback.format_exc())
+                    if not suppress_errors or isinstance(error, SystemExit):
+                        raise
+                    else:
+                        logger.error(error)
+                        return key, error if return_exception_objects else error_message
+                else:
+                    self.connection.commit_transaction()
+                    if jobs is not None:
+                        jobs.complete(self.target.table_name, self._job_key(key))
+                finally:
+                    self.__class__._allow_insert = False
 
     def progress(self, *restrictions, display=True):
         """
-        report progress of populating the table
-        :return: remaining, total -- tuples to be populated
+        Report the progress of populating the table.
+        :return: (remaining, total) -- numbers of tuples to be populated
         """
         todo = self._jobs_to_do(restrictions)
         total = len(todo)
@@ -193,5 +272,6 @@ def progress(self, *restrictions, display=True):
             print('%-20s' % self.__class__.__name__,
                   'Completed %d of %d (%2.1f%%)   %s' % (
                       total - remaining, total, 100 - 100 * remaining / (total+1e-12),
-                      datetime.datetime.strftime(datetime.datetime.now(), '%Y-%m-%d %H:%M:%S')), flush=True)
+                      datetime.datetime.strftime(datetime.datetime.now(),
+                                                 '%Y-%m-%d %H:%M:%S')), flush=True)
         return remaining, total
diff --git a/datajoint/blob.py b/datajoint/blob.py
index 225c68c1f..d3837cb6a 100644
--- a/datajoint/blob.py
+++ b/datajoint/blob.py
@@ -164,7 +164,9 @@ def pack_blob(self, obj):
             return self.pack_recarray(np.array(obj))
         if isinstance(obj, np.number):
             return self.pack_array(np.array(obj))
-        if isinstance(obj, (np.bool, np.bool_)):
+        if isinstance(obj, (bool, np.bool_)):
+            return self.pack_array(np.array(obj))
+        if isinstance(obj, (float, int, complex)):
             return self.pack_array(np.array(obj))
         if isinstance(obj, (datetime.datetime, datetime.date, datetime.time)):
             return self.pack_datetime(obj)
@@ -365,7 +367,7 @@ def read_struct(self):
         raw_data = [
             tuple(self.read_blob(n_bytes=int(self.read_value('uint64'))) for _ in range(n_fields))
             for __ in range(n_elem)]
-        data = np.array(raw_data, dtype=list(zip(field_names, repeat(np.object))))
+        data = np.array(raw_data, dtype=list(zip(field_names, repeat(object))))
         return self.squeeze(data.reshape(shape, order="F"), convert_to_scalar=False).view(MatStruct)
 
     def pack_struct(self, array):
diff --git a/datajoint/connection.py b/datajoint/connection.py
index cc4961adb..3daac4bac 100644
--- a/datajoint/connection.py
+++ b/datajoint/connection.py
@@ -278,7 +278,7 @@ def query(self, query, args=(), *, as_dict=False, suppress_warnings=True, reconn
         # check cache first:
         use_query_cache = bool(self._query_cache)
         if use_query_cache and not re.match(r"\s*(SELECT|SHOW)", query):
-            raise errors.DataJointError("Only SELECT query are allowed when query caching is on.")
+            raise errors.DataJointError("Only SELECT queries are allowed when query caching is on.")
         if use_query_cache:
             if not config['query_cache']:
                 raise errors.DataJointError("Provide filepath dj.config['query_cache'] when using query caching.")
diff --git a/datajoint/declare.py b/datajoint/declare.py
index e94c1bfef..93e7592de 100644
--- a/datajoint/declare.py
+++ b/datajoint/declare.py
@@ -11,7 +11,7 @@
 
 UUID_DATA_TYPE = 'binary(16)'
 MAX_TABLE_NAME_LENGTH = 64
-CONSTANT_LITERALS = {'CURRENT_TIMESTAMP'}  # SQL literals to be used without quotes (case insensitive)
+CONSTANT_LITERALS = {'CURRENT_TIMESTAMP', 'NULL'}  # SQL literals to be used without quotes (case insensitive)
 EXTERNAL_TABLE_ROOT = '~external'
 
 TYPE_PATTERN = {k: re.compile(v, re.I) for k, v in dict(
diff --git a/datajoint/dependencies.py b/datajoint/dependencies.py
index c1a331b7d..e5e94225d 100644
--- a/datajoint/dependencies.py
+++ b/datajoint/dependencies.py
@@ -18,7 +18,7 @@ def unite_master_parts(lst):
     """
     for i in range(2, len(lst)):
         name = lst[i]
-        match = re.match(r'(?P<master>`\w+`.`\w+)__\w+`', name)
+        match = re.match(r'(?P<master>`\w+`.`#?\w+)__\w+`', name)
         if match:  # name is a part table
             master = match.group('master')
             for j in range(i-1, -1, -1):
@@ -26,8 +26,6 @@ def unite_master_parts(lst):
                     # move from the ith position to the (j+1)th position
                     lst[j+1:i+1] = [name] + lst[j+1:i]
                     break
-            else:
-                raise DataJointError("Found a part table {name} without its master table.".format(name=name))
     return lst
 
 
diff --git a/datajoint/diagram.py b/datajoint/diagram.py
index c4f823035..fa03c123a 100644
--- a/datajoint/diagram.py
+++ b/datajoint/diagram.py
@@ -296,7 +296,7 @@ def make_dot(self):
                 node.set_style('filled')
 
             for edge in dot.get_edges():
-                # see http://www.graphviz.org/content/attrs
+                # see https://graphviz.org/doc/info/attrs.html
                 src = edge.get_source().strip('"')
                 dest = edge.get_destination().strip('"')
                 props = graph.get_edge_data(src, dest)
diff --git a/datajoint/expression.py b/datajoint/expression.py
index 507af14f3..624f1122b 100644
--- a/datajoint/expression.py
+++ b/datajoint/expression.py
@@ -9,6 +9,7 @@
 from .preview import preview, repr_html
 from .condition import AndList, Not, \
     make_condition, assert_join_compatibility, extract_column_names, PromiscuousOperand
+from .declare import CONSTANT_LITERALS
 
 logger = logging.getLogger(__name__)
 
@@ -36,7 +37,7 @@ class QueryExpression:
     """
     _restriction = None
     _restriction_attributes = None
-    _left = []  # True for left joins, False for inner joins
+    _left = []  # list of booleans True for left joins, False for inner joins
     _original_heading = None  # heading before projections
 
     # subclasses or instantiators must provide values
@@ -44,6 +45,9 @@ class QueryExpression:
     _heading = None
     _support = None
 
+    # If the query will be using distinct
+    _distinct = False
+
     @property
     def connection(self):
         """ a dj.Connection object """
@@ -98,7 +102,7 @@ def from_clause(self):
         return clause
 
     def where_clause(self):
-        return '' if not self.restriction else ' WHERE(%s)' % ')AND('.join(
+        return '' if not self.restriction else ' WHERE (%s)' % ')AND('.join(
             str(s) for s in self.restriction)
 
     def make_sql(self, fields=None):
@@ -106,9 +110,8 @@ def make_sql(self, fields=None):
         Make the SQL SELECT statement.
         :param fields: used to explicitly set the select attributes
         """
-        distinct = self.heading.names == self.primary_key
         return 'SELECT {distinct}{fields} FROM {from_}{where}'.format(
-            distinct="DISTINCT " if distinct else "",
+            distinct="DISTINCT " if self._distinct else "",
             fields=self.heading.as_sql(fields or self.heading.names),
             from_=self.from_clause(), where=self.where_clause())
 
@@ -260,15 +263,17 @@ def join(self, other, semantic_check=True, left=False):
         if semantic_check:
             assert_join_compatibility(self, other)
         join_attributes = set(n for n in self.heading.names if n in other.heading.names)
-        # needs subquery if FROM class has common attributes with the other's FROM clause
+        # needs subquery if self's FROM clause has common attributes with other's FROM clause
         need_subquery1 = need_subquery2 = bool(
             (set(self.original_heading.names) & set(other.original_heading.names))
             - join_attributes)
         # need subquery if any of the join attributes are derived
         need_subquery1 = (need_subquery1 or isinstance(self, Aggregation) or
-                          any(n in self.heading.new_attributes for n in join_attributes))
+                          any(n in self.heading.new_attributes for n in join_attributes)
+                          or isinstance(self, Union))
         need_subquery2 = (need_subquery2 or isinstance(other, Aggregation) or
-                          any(n in other.heading.new_attributes for n in join_attributes))
+                          any(n in other.heading.new_attributes for n in join_attributes)
+                          or isinstance(self, Union))
         if need_subquery1:
             self = self.make_subquery()
         if need_subquery2:
@@ -301,7 +306,7 @@ def proj(self, *attributes, **named_attributes):
         self.proj(...) or self.proj(Ellipsis) -- include all attributes (return self)
         self.proj() -- include only primary key
         self.proj('attr1', 'attr2')  -- include primary key and attributes attr1 and attr2
-        self.proj(..., '-attr1', '-attr2')  -- include attributes except attr1 and attr2
+        self.proj(..., '-attr1', '-attr2')  -- include all attributes except attr1 and attr2
         self.proj(name1='attr1') -- include primary key and 'attr1' renamed as name1
         self.proj('attr1', dup='(attr1)') -- include primary key and attribute attr1 twice, with the duplicate 'dup'
         self.proj(k='abs(attr1)') adds the new attribute k with the value computed as an expression (SQL syntax)
@@ -309,9 +314,9 @@ def proj(self, *attributes, **named_attributes):
         Each attribute name can only be used once.
         """
         # new attributes in parentheses are included again with the new name without removing original
-        duplication_pattern = re.compile(r'\s*\(\s*(?P<name>[a-z][a-z_0-9]*)\s*\)\s*$')
+        duplication_pattern = re.compile(fr'^\s*\(\s*(?!{"|".join(CONSTANT_LITERALS)})(?P<name>[a-zA-Z_]\w*)\s*\)\s*$')
         # attributes without parentheses renamed
-        rename_pattern = re.compile(r'\s*(?P<name>[a-z][a-z_0-9]*)\s*$')
+        rename_pattern = re.compile(fr'^\s*(?!{"|".join(CONSTANT_LITERALS)})(?P<name>[a-zA-Z_]\w*)\s*$')
         replicate_map = {k: m.group('name')
                          for k, m in ((k, duplication_pattern.match(v)) for k, v in named_attributes.items()) if m}
         rename_map = {k: m.group('name')
@@ -440,8 +445,10 @@ def tail(self, limit=25, **fetch_kwargs):
     def __len__(self):
         """:return: number of elements in the result set e.g. ``len(q1)``."""
         return self.connection.query(
-            'SELECT count(DISTINCT {fields}) FROM {from_}{where}'.format(
-                fields=self.heading.as_sql(self.primary_key, include_aliases=False),
+            'SELECT {select_} FROM {from_}{where}'.format(
+                select_=('count(*)' if any(self._left)
+                         else 'count(DISTINCT {fields})'.format(fields=self.heading.as_sql(
+                            self.primary_key, include_aliases=False))),
                 from_=self.from_clause(),
                 where=self.where_clause())).fetchone()[0]
 
@@ -554,7 +561,7 @@ def create(cls, arg, group, keep_all_rows=False):
         if inspect.isclass(group) and issubclass(group, QueryExpression):
             group = group()   # instantiate if a class
         assert isinstance(group, QueryExpression)
-        if keep_all_rows and len(group.support) > 1:
+        if keep_all_rows and len(group.support) > 1 or group.heading.new_attributes:
             group = group.make_subquery()  # subquery if left joining a join
         join = arg.join(group, left=keep_all_rows)  # reuse the join logic
         result = cls()
@@ -599,6 +606,8 @@ class Union(QueryExpression):
     """
     Union is the private DataJoint class that implements the union operator.
     """
+    __count = count()
+
     @classmethod
     def create(cls, arg1, arg2):
         if inspect.isclass(arg2) and issubclass(arg2, QueryExpression):
@@ -625,9 +634,11 @@ def make_sql(self):
         if not arg1.heading.secondary_attributes and not arg2.heading.secondary_attributes:
             # no secondary attributes: use UNION DISTINCT
             fields = arg1.primary_key
-            return "({sql1}) UNION ({sql2})".format(
-                sql1=arg1.make_sql(fields),
-                sql2=arg2.make_sql(fields))
+            return ("SELECT * FROM (({sql1}) UNION ({sql2})) as `_u{alias}`".format(
+                sql1=arg1.make_sql() if isinstance(arg1, Union) else arg1.make_sql(fields),
+                sql2=arg2.make_sql() if isinstance(arg2, Union) else arg2.make_sql(fields),
+                alias=next(self.__count)
+                ))
         # with secondary attributes, use union of left join with antijoin
         fields = self.heading.names
         sql1 = arg1.join(arg2, left=True).make_sql(fields)
@@ -718,6 +729,7 @@ def __and__(self, other):
         if not isinstance(other, QueryExpression):
             raise DataJointError('Set U can only be restricted with a QueryExpression.')
         result = copy.copy(other)
+        result._distinct = True
         result._heading = result.heading.set_primary_key(self.primary_key)
         result = result.proj()
         return result
diff --git a/datajoint/external.py b/datajoint/external.py
index 808b2d65b..62120e306 100644
--- a/datajoint/external.py
+++ b/datajoint/external.py
@@ -1,5 +1,5 @@
 from pathlib import Path, PurePosixPath, PureWindowsPath
-from collections import Mapping
+from collections.abc import Mapping
 from tqdm import tqdm
 from .settings import config
 from .errors import DataJointError, MissingExternalFile
@@ -127,7 +127,10 @@ def _remove_external_file(self, external_path):
         if self.spec['protocol'] == 's3':
             self.s3.remove_object(external_path)
         elif self.spec['protocol'] == 'file':
-            Path(external_path).unlink()
+            try:
+                Path(external_path).unlink()
+            except FileNotFoundError:
+                pass
 
     def exists(self, external_filepath):
         """
@@ -314,11 +317,12 @@ def used(self):
         return self & [FreeTable(self.connection, ref['referencing_table']).proj(hash=ref['column_name'])
                        for ref in self.references]
 
-    def delete(self, *, delete_external_files=None, limit=None, display_progress=True):
+    def delete(self, *, delete_external_files=None, limit=None, display_progress=True, errors_as_string=True):
         """
         :param delete_external_files: True or False. If False, only the tracking info is removed from the
         external store table but the external files remain intact. If True, then the external files
         themselves are deleted too.
+        :param errors_as_string: If True any errors returned when deleting from external files will be strings
         :param limit: (integer) limit the number of items to delete
         :param display_progress: if True, display progress as files are cleaned up
         :return: if deleting external files, returns errors
@@ -337,16 +341,20 @@ def delete(self, *, delete_external_files=None, limit=None, display_progress=Tru
             # delete items one by one, close to transaction-safe
             error_list = []
             for uuid, external_path in items:
-                try:
-                    count = (self & {'hash': uuid}).delete_quick(get_count=True)  # optimize
-                except Exception:
-                    pass   # if delete failed, do not remove the external file
-                else:
-                    assert count in (0, 1)
+                row = (self & {'hash': uuid}).fetch()
+                if row.size:
                     try:
-                        self._remove_external_file(external_path)
-                    except Exception as error:
-                        error_list.append((uuid, external_path, str(error)))
+                        (self & {'hash': uuid}).delete_quick()
+                    except Exception:
+                        pass   # if delete failed, do not remove the external file
+                    else:
+                        try:
+                            self._remove_external_file(external_path)
+                        except Exception as error:
+                            # adding row back into table after failed delete
+                            self.insert1(row[0], skip_duplicates=True)
+                            error_list.append((uuid, external_path,
+                                               str(error) if errors_as_string else error))
             return error_list
 
 
diff --git a/datajoint/jobs.py b/datajoint/jobs.py
index 97433382d..571270931 100644
--- a/datajoint/jobs.py
+++ b/datajoint/jobs.py
@@ -32,7 +32,7 @@ def __init__(self, conn, database):
         status  :enum('reserved','error','ignore')  # if tuple is missing, the job is available
         key=null  :blob  # structure containing the key
         error_message=""  :varchar({error_message_length})  # error message returned if failed
-        error_stack=null  :blob  # error stack if failed
+        error_stack=null  :mediumblob  # error stack if failed
         user="" :varchar(255) # database user
         host=""  :varchar(255)  # system hostname
         pid=0  :int unsigned  # system process id
diff --git a/datajoint/s3.py b/datajoint/s3.py
index f50e479db..e26cf1329 100644
--- a/datajoint/s3.py
+++ b/datajoint/s3.py
@@ -3,6 +3,7 @@
 """
 from io import BytesIO
 import minio   # https://docs.minio.io/docs/python-client-api-reference
+import urllib3
 import warnings
 import uuid
 import logging
@@ -16,9 +17,24 @@ class Folder:
     """
     A Folder instance manipulates a flat folder of objects within an S3-compatible object store
     """
-    def __init__(self, endpoint, bucket, access_key, secret_key, *, secure=False, **_):
-        self.client = minio.Minio(endpoint, access_key=access_key, secret_key=secret_key,
-                                  secure=secure)
+    def __init__(self, endpoint, bucket, access_key, secret_key, *, secure=False,
+                 proxy_server=None, **_):
+        # from https://docs.min.io/docs/python-client-api-reference
+        self.client = minio.Minio(
+            endpoint,
+            access_key=access_key,
+            secret_key=secret_key,
+            secure=secure,
+            http_client=(
+                urllib3.ProxyManager(proxy_server,
+                                     timeout=urllib3.Timeout.DEFAULT_TIMEOUT,
+                                     cert_reqs="CERT_REQUIRED",
+                                     retries=urllib3.Retry(total=5,
+                                                           backoff_factor=0.2,
+                                                           status_forcelist=[500, 502, 503,
+                                                                             504]))
+                if proxy_server else None),
+            )
         self.bucket = bucket
         if not self.client.bucket_exists(bucket):
             raise errors.BucketInaccessible('Inaccessible s3 bucket %s' % bucket)
@@ -76,12 +92,11 @@ def get_size(self, name):
         except minio.error.S3Error as e:
             if e.code == 'NoSuchKey':
                 raise errors.MissingExternalFile
-            else:
-                raise e
+            raise e
 
     def remove_object(self, name):
         logger.debug('remove_object: {}:{}'.format(self.bucket, name))
         try:
             self.client.remove_object(self.bucket, str(name))
-        except minio.ResponseError:
-            return errors.DataJointError('Failed to delete %s from s3 storage' % name)
+        except minio.error.MinioException:
+            raise errors.DataJointError('Failed to delete %s from s3 storage' % name)
diff --git a/datajoint/schemas.py b/datajoint/schemas.py
index d108caef2..ab2fc03af 100644
--- a/datajoint/schemas.py
+++ b/datajoint/schemas.py
@@ -23,7 +23,6 @@ def ordered_dir(class_):
     """
     List (most) attributes of the class including inherited ones, similar to `dir` build-in function,
     but respects order of attribute declaration as much as possible.
-    This becomes unnecessary in Python 3.6+ as dicts became ordered.
     :param class_: class to list members for
     :return: a list of attributes declared in class_ and its superclasses
     """
@@ -186,6 +185,7 @@ def _decorate_table(self, table_class, context, assert_declared=False):
             if not self.create_tables or assert_declared:
                 raise DataJointError('Table `%s` not declared' % instance.table_name)
             instance.declare(context)
+            self.connection.dependencies.clear()
         is_declared = is_declared or instance.is_declared
 
         # add table definition to the doc string
diff --git a/datajoint/settings.py b/datajoint/settings.py
index 3ebcf3ed9..75f265f87 100644
--- a/datajoint/settings.py
+++ b/datajoint/settings.py
@@ -58,7 +58,7 @@
 }
 
 
-class Config(collections.MutableMapping):
+class Config(collections.abc.MutableMapping):
 
     instance = None
 
@@ -137,7 +137,8 @@ def get_store_spec(self, store):
         spec['subfolding'] = spec.get('subfolding', DEFAULT_SUBFOLDING)
         spec_keys = {  # REQUIRED in uppercase and allowed in lowercase
             'file': ('PROTOCOL', 'LOCATION', 'subfolding', 'stage'),
-            's3': ('PROTOCOL', 'ENDPOINT', 'BUCKET', 'ACCESS_KEY', 'SECRET_KEY', 'LOCATION', 'secure', 'subfolding', 'stage')}
+            's3': ('PROTOCOL', 'ENDPOINT', 'BUCKET', 'ACCESS_KEY', 'SECRET_KEY', 'LOCATION',
+                   'secure', 'subfolding', 'stage', 'proxy_server')}
 
         try:
             spec_keys = spec_keys[spec.get('protocol', '').lower()]
diff --git a/datajoint/table.py b/datajoint/table.py
index ed4dcfa4f..587de6aef 100644
--- a/datajoint/table.py
+++ b/datajoint/table.py
@@ -14,7 +14,7 @@
 from .condition import make_condition
 from .expression import QueryExpression
 from . import blob
-from .utils import user_choice
+from .utils import user_choice, get_master
 from .heading import Heading
 from .errors import (DuplicateError, AccessError, DataJointError, UnknownAttributeError,
                      IntegrityError)
@@ -46,16 +46,17 @@ class _RenameMap(tuple):
 
 class Table(QueryExpression):
     """
-    Table is an abstract class that represents a base relation, i.e. a table in the schema.
+    Table is an abstract class that represents a table in the schema.
+    It implements insert and delete methods and inherits query functionality.
     To make it a concrete class, override the abstract properties specifying the connection,
     table name, database, and definition.
-    A Relation implements insert and delete methods in addition to inherited relational operators.
     """
 
     _table_name = None  # must be defined in subclass
     _log_ = None  # placeholder for the Log table object
 
-    # These properties must be set by the schema decorator (schemas.py) at class level or by FreeTable at instance level
+    # These properties must be set by the schema decorator (schemas.py) at class level
+    # or by FreeTable at instance level
     database = None
     declaration_context = None
 
@@ -65,7 +66,8 @@ def table_name(self):
 
     @property
     def definition(self):
-        raise NotImplementedError('Subclasses of Table must implement the `definition` property')
+        raise NotImplementedError(
+            'Subclasses of Table must implement the `definition` property')
 
     def declare(self, context=None):
         """
@@ -95,7 +97,8 @@ def alter(self, prompt=True, context=None):
         """
         if self.connection.in_transaction:
             raise DataJointError(
-                'Cannot update table declaration inside a transaction, e.g. from inside a populate/make call')
+                'Cannot update table declaration inside a transaction, '
+                'e.g. from inside a populate/make call')
         if context is None:
             frame = inspect.currentframe().f_back
             context = dict(frame.f_globals, **frame.f_locals)
@@ -117,7 +120,8 @@ def alter(self, prompt=True, context=None):
                     # skip if no create privilege
                     pass
                 else:
-                    self.__class__._heading = Heading(table_info=self.heading.table_info)  # reset heading
+                    # reset heading
+                    self.__class__._heading = Heading(table_info=self.heading.table_info)
                     if prompt:
                         print('Table altered')
                     self._log('Altered ' + self.full_table_name)
@@ -226,9 +230,12 @@ def external(self):
     def update1(self, row):
         """
         update1 updates one existing entry in the table.
-        Caution: Updates are not part of the DataJoint data manipulation model. For strict data integrity,
-        use delete and insert.
-        :param row: a dict containing the primary key and the attributes to update.
+        Caution: In DataJoint the primary modes for data manipulation is to ``insert`` and ``delete``
+        entire records since referential integrity works on the level of records, not fields.
+        Therefore, updates are reserved for corrective operations outside of main workflow.
+        Use UPDATE methods sparingly with full awareness of potential violations of assumptions.
+
+        :param row: a ``dict`` containing the primary key values and the attributes to update.
         Setting an attribute value to None will reset it to the default value (if any)
         The primary key attributes must always be provided.
         Examples:
@@ -241,7 +248,8 @@ def update1(self, row):
         if not set(row).issuperset(self.primary_key):
             raise DataJointError('The argument of update1 must supply all primary key values.')
         try:
-            raise DataJointError('Attribute `%s` not found.' % next(k for k in row if k not in self.heading.names))
+            raise DataJointError('Attribute `%s` not found.' %
+                                 next(k for k in row if k not in self.heading.names))
         except StopIteration:
             pass  # ok
         if len(self.restriction):
@@ -250,7 +258,8 @@ def update1(self, row):
         if len(self & key) != 1:
             raise DataJointError('Update entry must exist.')
         # UPDATE query
-        row = [self.__make_placeholder(k, v) for k, v in row.items() if k not in self.primary_key]
+        row = [self.__make_placeholder(k, v) for k, v in row.items()
+               if k not in self.primary_key]
         query = "UPDATE {table} SET {assignments} WHERE {where}".format(
             table=self.full_table_name,
             assignments=",".join('`%s`=%s' % r[:2] for r in row),
@@ -259,22 +268,24 @@ def update1(self, row):
 
     def insert1(self, row, **kwargs):
         """
-        Insert one data record or one Mapping (like a dict).
-        :param row: a numpy record, a dict-like object, or an ordered sequence to be inserted as one row.
-        For kwargs, see insert()
+        Insert one data record into the table. For ``kwargs``, see ``insert()``.
+
+        :param row: a numpy record, a dict-like object, or an ordered sequence to be inserted
+        as one row.
         """
         self.insert((row,), **kwargs)
 
-    def insert(self, rows, replace=False, skip_duplicates=False, ignore_extra_fields=False, allow_direct_insert=None):
+    def insert(self, rows, replace=False, skip_duplicates=False, ignore_extra_fields=False,
+               allow_direct_insert=None):
         """
         Insert a collection of rows.
-        :param rows: An iterable where an element is a numpy record, a dict-like object, a pandas.DataFrame, a sequence,
-            or a query expression with the same heading as table self.
+        :param rows: An iterable where an element is a numpy record, a dict-like object, a
+            pandas.DataFrame, a sequence, or a query expression with the same heading as self.
         :param replace: If True, replaces the existing tuple.
         :param skip_duplicates: If True, silently skip duplicate inserts.
         :param ignore_extra_fields: If False, fields that are not in the heading raise error.
         :param allow_direct_insert: applies only in auto-populated tables.
-                                    If False (default), insert are allowed only from inside the make callback.
+    If False (default), insert are allowed only from inside the make callback.
         Example::
         >>> relation.insert([
         >>>     dict(subject_id=7, species="mouse", date_of_birth="2014-09-01"),
@@ -288,20 +299,22 @@ def insert(self, rows, replace=False, skip_duplicates=False, ignore_extra_fields
             ).to_records(index=False)
 
         # prohibit direct inserts into auto-populated tables
-        if not allow_direct_insert and not getattr(self, '_allow_insert', True):  # allow_insert is only used in AutoPopulate
+        if not allow_direct_insert and not getattr(self, '_allow_insert', True):
             raise DataJointError(
-                'Inserts into an auto-populated table can only done inside its make method during a populate call.'
+                'Inserts into an auto-populated table can only be done inside '
+                'its make method during a populate call.'
                 ' To override, set keyword argument allow_direct_insert=True.')
 
-        if inspect.isclass(rows) and issubclass(rows, QueryExpression):   # instantiate if a class
-            rows = rows()
+        if inspect.isclass(rows) and issubclass(rows, QueryExpression):
+            rows = rows()  # instantiate if a class
         if isinstance(rows, QueryExpression):
             # insert from select
             if not ignore_extra_fields:
                 try:
                     raise DataJointError(
-                        "Attribute %s not found. To ignore extra attributes in insert, set ignore_extra_fields=True." %
-                        next(name for name in rows.heading if name not in self.heading))
+                        "Attribute %s not found. To ignore extra attributes in insert, "
+                        "set ignore_extra_fields=True." % next(
+                            name for name in rows.heading if name not in self.heading))
                 except StopIteration:
                     pass
             fields = list(name for name in rows.heading if name in self.heading)
@@ -348,58 +361,65 @@ def delete_quick(self, get_count=False):
         self._log(query[:255])
         return count
 
-    def _delete_cascade(self):
-        """service function to perform cascading deletes recursively."""
-        max_attempts = 50
-        for _ in range(max_attempts):
-            try:
-                delete_count = self.delete_quick(get_count=True)
-            except IntegrityError as error:
-                match = foreign_key_error_regexp.match(error.args[0]).groupdict()
-                if "`.`" not in match['child']:  # if schema name missing, use self
-                    match['child'] = '{}.{}'.format(self.full_table_name.split(".")[0],
-                                                    match['child'])
-                if match['pk_attrs'] is not None:  # fully matched, adjusting the keys
-                    match['fk_attrs'] = [k.strip('`') for k in match['fk_attrs'].split(',')]
-                    match['pk_attrs'] = [k.strip('`') for k in match['pk_attrs'].split(',')]
-                else:  # only partially matched, querying with constraint to determine keys
-                    match['fk_attrs'], match['parent'], match['pk_attrs'] = list(map(
-                        list, zip(*self.connection.query(constraint_info_query, args=(
-                            match['name'].strip('`'),
-                            *[_.strip('`') for _ in match['child'].split('`.`')]
-                            )).fetchall())))
-                    match['parent'] = match['parent'][0]
-                # restrict child by self if
-                # 1. if self's restriction attributes are not in child's primary key
-                # 2. if child renames any attributes
-                # otherwise restrict child by self's restriction.
-                child = FreeTable(self.connection, match['child'])
-                if set(self.restriction_attributes) <= set(child.primary_key) and \
-                        match['fk_attrs'] == match['pk_attrs']:
-                    child._restriction = self._restriction
-                elif match['fk_attrs'] != match['pk_attrs']:
-                    child &= self.proj(**dict(zip(match['fk_attrs'],
-                                                  match['pk_attrs'])))
-                else:
-                    child &= self.proj()
-                child._delete_cascade()
-            else:
-                print("Deleting {count} rows from {table}".format(
-                    count=delete_count, table=self.full_table_name))
-                break
-        else:
-            raise DataJointError('Exceeded maximum number of delete attempts.')
-        return delete_count
-
-    def delete(self, transaction=True, safemode=None):
+    def delete(self, transaction=True, safemode=None, force_parts=False):
         """
         Deletes the contents of the table and its dependent tables, recursively.
 
         :param transaction: if True, use the entire delete becomes an atomic transaction.
+            This is the default and recommended behavior. Set to False if this delete is nested
+            within another transaction.
         :param safemode: If True, prohibit nested transactions and prompt to confirm. Default
             is dj.config['safemode'].
+        :param force_parts: Delete from parts even when not deleting from their masters.
         :return: number of deleted rows (excluding those from dependent tables)
         """
+        deleted = set()
+
+        def cascade(table):
+            """service function to perform cascading deletes recursively."""
+            max_attempts = 50
+            for _ in range(max_attempts):
+                try:
+                    delete_count = table.delete_quick(get_count=True)
+                except IntegrityError as error:
+                    match = foreign_key_error_regexp.match(error.args[0]).groupdict()
+                    if "`.`" not in match['child']:  # if schema name missing, use table
+                        match['child'] = '{}.{}'.format(table.full_table_name.split(".")[0],
+                                                        match['child'])
+                    if match['pk_attrs'] is not None:  # fully matched, adjusting the keys
+                        match['fk_attrs'] = [k.strip('`') for k in match['fk_attrs'].split(',')]
+                        match['pk_attrs'] = [k.strip('`') for k in match['pk_attrs'].split(',')]
+                    else:  # only partially matched, querying with constraint to determine keys
+                        match['fk_attrs'], match['parent'], match['pk_attrs'] = list(map(
+                            list, zip(*table.connection.query(constraint_info_query, args=(
+                                match['name'].strip('`'),
+                                *[_.strip('`') for _ in match['child'].split('`.`')]
+                            )).fetchall())))
+                        match['parent'] = match['parent'][0]
+
+                    # Restrict child by table if
+                    #   1. if table's restriction attributes are not in child's primary key
+                    #   2. if child renames any attributes
+                    # Otherwise restrict child by table's restriction.
+                    child = FreeTable(table.connection, match['child'])
+                    if set(table.restriction_attributes) <= set(child.primary_key) and \
+                            match['fk_attrs'] == match['pk_attrs']:
+                        child._restriction = table._restriction
+                    elif match['fk_attrs'] != match['pk_attrs']:
+                        child &= table.proj(**dict(zip(match['fk_attrs'],
+                                                       match['pk_attrs'])))
+                    else:
+                        child &= table.proj()
+                    cascade(child)
+                else:
+                    deleted.add(table.full_table_name)
+                    print("Deleting {count} rows from {table}".format(
+                        count=delete_count, table=table.full_table_name))
+                    break
+            else:
+                raise DataJointError('Exceeded maximum number of delete attempts.')
+            return delete_count
+
         safemode = config['safemode'] if safemode is None else safemode
 
         # Start transaction
@@ -416,12 +436,23 @@ def delete(self, transaction=True, safemode=None):
 
         # Cascading delete
         try:
-            delete_count = self._delete_cascade()
+            delete_count = cascade(self)
         except:
             if transaction:
                 self.connection.cancel_transaction()
             raise
 
+        if not force_parts:
+            # Avoid deleting from child before master (See issue #151)
+            for part in deleted:
+                master = get_master(part)
+                if master and master not in deleted:
+                    if transaction:
+                        self.connection.cancel_transaction()
+                    raise DataJointError(
+                        'Attempt to delete part table {part} before deleting from '
+                        'its master {master} first.'.format(part=part, master=master))
+
         # Confirm and commit
         if delete_count == 0:
             if safemode:
@@ -460,12 +491,21 @@ def drop(self):
         User is prompted for confirmation if config['safemode'] is set to True.
         """
         if self.restriction:
-            raise DataJointError('A relation with an applied restriction condition cannot be dropped.'
+            raise DataJointError('A table with an applied restriction cannot be dropped.'
                                  ' Call drop() on the unrestricted Table.')
         self.connection.dependencies.load()
         do_drop = True
-        tables = [table for table in self.connection.dependencies.descendants(self.full_table_name)
-                  if not table.isdigit()]
+        tables = [table for table in self.connection.dependencies.descendants(
+            self.full_table_name) if not table.isdigit()]
+
+        # avoid dropping part tables without their masters: See issue #374
+        for part in tables:
+            master = get_master(part)
+            if master and master not in tables:
+                raise DataJointError(
+                    'Attempt to drop part table {part} before dropping '
+                    'its master. Drop {master} first.'.format(part=part, master=master))
+
         if config['safemode']:
             for table in tables:
                 print(table, '(%d tuples)' % len(FreeTable(self.connection, table)))
@@ -555,12 +595,14 @@ def describe(self, context=None, printout=True):
 
     def _update(self, attrname, value=None):
         """
-            This is a deprecated function to be removed in datajoint 0.14. Use .update1 instead.
+            This is a deprecated function to be removed in datajoint 0.14.
+            Use ``.update1`` instead.
 
-            Updates a field in an existing tuple. This is not a datajoyous operation and should not be used
-            routinely. Relational database maintain referential integrity on the level of a tuple. Therefore,
-            the UPDATE operator can violate referential integrity. The datajoyous way to update information is
-            to delete the entire tuple and insert the entire update tuple.
+            Updates a field in one existing tuple. self must be restricted to exactly one entry.
+            In DataJoint the principal way of updating data is to delete and re-insert the
+            entire record and updates are reserved for corrective actions.
+            This is because referential integrity is observed on the level of entire
+            records rather than individual attributes.
 
             Safety constraints:
                1. self must be restricted to exactly one tuple
@@ -586,7 +628,7 @@ def _update(self, attrname, value=None):
             value = blob.pack(value)
             placeholder = '%s'
         elif attr.numeric:
-            if value is None or np.isnan(np.float(value)):  # nans are turned into NULLs
+            if value is None or np.isnan(float(value)):  # nans are turned into NULLs
                 placeholder = 'NULL'
                 value = None
             else:
@@ -615,7 +657,7 @@ def __make_placeholder(self, name, value, ignore_extra_fields=False):
         attr = self.heading[name]
         if attr.adapter:
             value = attr.adapter.put(value)
-        if value is None or (attr.numeric and (value == '' or np.isnan(np.float(value)))):
+        if value is None or (attr.numeric and (value == '' or np.isnan(float(value)))):
             # set default value
             placeholder, value = 'DEFAULT', None
         else:  # not NULL
@@ -664,7 +706,7 @@ def check_fields(fields):
                         if field not in self.heading:
                             raise KeyError(u'`{0:s}` is not in the table heading'.format(field))
             elif set(field_list) != set(fields).intersection(self.heading.names):
-                raise DataJointError('Attempt to insert rows with different fields')
+                raise DataJointError('Attempt to insert rows with different fields.')
 
         if isinstance(row, np.void):  # np.array
             check_fields(row.dtype.fields)
diff --git a/datajoint/user_tables.py b/datajoint/user_tables.py
index b7bfd9db2..76cea5fbf 100644
--- a/datajoint/user_tables.py
+++ b/datajoint/user_tables.py
@@ -173,9 +173,10 @@ def delete(self, force=False):
         unless force is True, prohibits direct deletes from parts.
         """
         if force:
-            super().delete()
+            super().delete(force_parts=True)
         else:
-            raise DataJointError('Cannot delete from a Part directly.  Delete from master instead')
+            raise DataJointError(
+                'Cannot delete from a Part directly. Delete from master instead')
 
     def drop(self, force=False):
         """
diff --git a/datajoint/utils.py b/datajoint/utils.py
index 42b18222b..057e7e820 100644
--- a/datajoint/utils.py
+++ b/datajoint/utils.py
@@ -31,6 +31,25 @@ def user_choice(prompt, choices=("yes", "no"), default=None):
     return response
 
 
+def get_master(full_table_name: str) -> str:
+    """
+    If the table name is that of a part table, then return what the master table name would be.
+    This follows DataJoint's table naming convention where a master and a part must be in the
+    same schema and the part table is prefixed with the master table name + ``__``.
+
+    Example:
+       `ephys`.`session`    -- master
+       `ephys`.`session__recording`  -- part
+
+    :param full_table_name: Full table name including part.
+    :type full_table_name: str
+    :return: Supposed master full table name or empty string if not a part table name.
+    :rtype: str
+    """
+    match = re.match(r'(?P<master>`\w+`.`\w+)__(?P<part>\w+)`', full_table_name)
+    return match['master'] + '`' if match else ''
+
+
 def to_camel_case(s):
     """
     Convert names with under score (_) separation into camel case names.
diff --git a/datajoint/version.py b/datajoint/version.py
index c5a4541fe..6253805eb 100644
--- a/datajoint/version.py
+++ b/datajoint/version.py
@@ -1,3 +1,3 @@
-__version__ = "0.13.2"
+__version__ = "0.13.3"
 
 assert len(__version__) <= 10  # The log table limits version to the 10 characters
diff --git a/docs-parts/computation/01-autopopulate_lang1.rst b/docs-parts/computation/01-autopopulate_lang1.rst
index 31cf621ab..376feb471 100644
--- a/docs-parts/computation/01-autopopulate_lang1.rst
+++ b/docs-parts/computation/01-autopopulate_lang1.rst
@@ -13,6 +13,6 @@
         def make(self, key):
             img = (test.Image & key).fetch1('image')
             key['filtered_image'] = myfilter(img)
-            self.insert(key)
+            self.insert1(key)
 
 The ``make`` method receives one argument: the dict ``key`` containing the primary key value of an element of :ref:`key source <keysource>` to be worked on.  
diff --git a/docs-parts/definition/11-ERD_lang1.rst b/docs-parts/definition/11-ERD_lang1.rst
index 1d3330cd2..647e77f04 100644
--- a/docs-parts/definition/11-ERD_lang1.rst
+++ b/docs-parts/definition/11-ERD_lang1.rst
@@ -1,11 +1,11 @@
 
-To plot the ERD for an entire schema, an ERD object can be initialized with the schema object (which is normally used to decorate table objects)
+To plot the Diagram for an entire schema, an Diagram object can be initialized with the schema object (which is normally used to decorate table objects)
 
 .. code-block:: python
 
     import datajoint as dj
     schema = dj.Schema('my_database')
-    dj.ERD(schema).draw()
+    dj.Diagram(schema).draw()
 
 or alternatively an object that has the schema object as an attribute, such as the module defining a schema:
 
@@ -13,9 +13,9 @@ or alternatively an object that has the schema object as an attribute, such as t
 
     import datajoint as dj
     import seq    # import the sequence module defining the seq database
-    dj.ERD(seq).draw()   # draw the ERD
+    dj.Diagram(seq).draw()   # draw the Diagram
 
 Note that calling the ``.draw()`` method is not necessary when working in a Jupyter notebook.
-The preferred workflow is to simply let the object display itself, for example by writing ``dj.ERD(seq)``.
-The ERD will then render in the notebook using its ``_repr_html_`` method.
-An ERD displayed without ``.draw()`` will be rendered as an SVG, and hovering the mouse over a table will reveal a compact version of the output of the ``.describe()`` method.
+You can simply let the object display itself, for example by entering ``dj.Diagram(seq)`` in a notebook cell.
+The Diagram will automatically render in the notebook by calling its ``_repr_html_`` method.
+A Diagram displayed without ``.draw()`` will be rendered as an SVG, and hovering the mouse over a table will reveal a compact version of the output of the ``.describe()`` method.
diff --git a/docs-parts/definition/11-ERD_lang2.rst b/docs-parts/definition/11-ERD_lang2.rst
index cb345a955..523b072fa 100644
--- a/docs-parts/definition/11-ERD_lang2.rst
+++ b/docs-parts/definition/11-ERD_lang2.rst
@@ -1,4 +1,4 @@
 
 .. code-block:: python
 
-    dj.ERD(seq.Genome).draw()
+    dj.Diagram(seq.Genome).draw()
diff --git a/docs-parts/definition/11-ERD_lang3.rst b/docs-parts/definition/11-ERD_lang3.rst
index 3474dcd45..91632fafb 100644
--- a/docs-parts/definition/11-ERD_lang3.rst
+++ b/docs-parts/definition/11-ERD_lang3.rst
@@ -1,5 +1,5 @@
 
 .. code-block:: python
 
-    # plot the ERD with tables Genome and Species from module seq.
-    (dj.ERD(seq.Genome) + dj.ERD(seq.Species)).draw()
+    # plot the Diagram with tables Genome and Species from module seq.
+    (dj.Diagram(seq.Genome) + dj.Diagram(seq.Species)).draw()
diff --git a/docs-parts/definition/11-ERD_lang4.rst b/docs-parts/definition/11-ERD_lang4.rst
index f67b40d10..03936a728 100644
--- a/docs-parts/definition/11-ERD_lang4.rst
+++ b/docs-parts/definition/11-ERD_lang4.rst
@@ -2,14 +2,14 @@
 .. code-block:: python
 
     # Plot all the tables directly downstream from ``seq.Genome``:
-    (dj.ERD(seq.Genome)+1).draw()
+    (dj.Diagram(seq.Genome)+1).draw()
 
 .. code-block:: python
 
     # Plot all the tables directly upstream from ``seq.Genome``:
-    (dj.ERD(seq.Genome)-1).draw()
+    (dj.Diagram(seq.Genome)-1).draw()
 
 .. code-block:: python
 
     # Plot the local neighborhood of ``seq.Genome``
-    (dj.ERD(seq.Genome)+1-1+1-1).draw()
+    (dj.Diagram(seq.Genome)+1-1+1-1).draw()
diff --git a/docs-parts/existing/1-Loading-Classes_lang1.rst b/docs-parts/existing/1-Loading-Classes_lang1.rst
index 70d7606f5..7d209b40c 100644
--- a/docs-parts/existing/1-Loading-Classes_lang1.rst
+++ b/docs-parts/existing/1-Loading-Classes_lang1.rst
@@ -70,7 +70,7 @@ not have classes declared to interact with it.
 
 So let's start over in this scenario.
 
-You can may use the ``dj.list_schemas`` function (new in DataJoint 0.12.0) to
+You may use the ``dj.list_schemas`` function (new in DataJoint 0.12.0) to
 list the names of database schemas available to you.
 
 .. code-block:: python
@@ -112,13 +112,13 @@ these tables. A similar situation arises when another developer has added new
 tables to the schema but has not yet shared the updated module code with you.
 Then the diagram will show a mixture of class names and database table names.
 
-Now you may use the ``schema.spawn_missing_classes`` method to spawn classes into
+Now you may use the ``spawn_missing_classes`` method to spawn classes into
 the local namespace for any tables missing their classes:
 
 .. code-block:: python
 
     schema.spawn_missing_classes()
-    dj.Di(schema)
+    dj.Diagram(schema)
 
 .. figure:: spawned-classes-ERD.svg
    :align: center
@@ -165,7 +165,7 @@ the table classes.
 
 .. code-block:: python
 
-    dj.Di(uni)
+    dj.Diagram(uni)
 
 .. figure:: added-example-ERD.svg
    :align: center
@@ -229,7 +229,7 @@ decorator for declaring new tables:
 
 .. code-block:: python
 
-    dj.Di(uni)
+    dj.Diagram(uni)
 
 .. figure:: added-example-ERD.svg
    :align: center
diff --git a/docs-parts/intro/Releases_lang1.rst b/docs-parts/intro/Releases_lang1.rst
index cc5acdb98..286096437 100644
--- a/docs-parts/intro/Releases_lang1.rst
+++ b/docs-parts/intro/Releases_lang1.rst
@@ -1,3 +1,23 @@
+0.13.3 -- Feb 9, 2022
+----------------------
+* Bugfix - Fix error in listing ancestors, descendants with part tables.
+* Bugfix - Fix Python 3.10 compatibility (#983) PR #972
+* Bugfix - Allow renaming non-conforming attributes in proj (#982) PR #972
+* Add - Expose proxy feature for S3 external stores (#961) PR #962
+* Add - implement multiprocessing in populate (#695) PR #704, #969
+* Bugfix - Dependencies not properly loaded on populate. (#902) PR #919
+* Bugfix - Replace use of numpy aliases of built-in types with built-in type. (#938) PR #939
+* Bugfix - Deletes and drops must include the master of each part. (#151, #374) PR #957
+* Bugfix - `ExternalTable.delete` should not remove row on error (#953) PR #956
+* Bugfix - Fix error handling of remove_object function in `s3.py` (#952) PR #955
+* Bugfix - Fix sql code generation to comply with sql mode ``ONLY_FULL_GROUP_BY`` (#916) PR #965
+* Bugfix - Fix count for left-joined ``QueryExpressions`` (#951) PR #966
+* Bugfix - Fix assertion error when performing a union into a join (#930) PR #967
+* Bugfix - Fix regression issue with `DISTINCT` clause and `GROUP_BY` (#914) PR #963
+* Update `~jobs.error_stack` from blob to mediumblob to allow error stacks >64kB in jobs (#984) PR #986
+* Bugfix - Fix error when performing a union on multiple tables (#926) PR #964
+* Add - Allow optional keyword arguments for `make()` in `populate()` PR #971
+
 0.13.2 -- May 7, 2021
 ----------------------
 * Update `setuptools_certificate` dependency to new name `otumat`
@@ -44,14 +64,14 @@
 * Fix display of part tables in `schema.save`. (#821) PR #833
 * Add `schema.list_tables`. (#838) PR #844
 * Fix minio new version regression.  PR #847
-* Add more S3 logging for debugging. (#831) PR #832 
+* Add more S3 logging for debugging. (#831) PR #832
 * Convert testing framework from TravisCI to GitHub Actions (#841) PR #840
 
 0.12.7 -- Oct 27, 2020
 ----------------------
 * Fix case sensitivity issues to adapt to MySQL 8+.  PR #819
 * Fix pymysql regression bug (#814) PR #816
-* Adapted attribute types now have `dtype=object` in all recarray results. PR #811 
+* Adapted attribute types now have `dtype=object` in all recarray results. PR #811
 
 0.12.6 -- May 15, 2020
 ----------------------
@@ -125,7 +145,7 @@
 
 0.11.1 -- Nov 15, 2018
 ----------------------
-* Fix ordering of attributes in proj (#483 and #516)
+* Fix ordering of attributes in proj (#483, #516)
 * Prohibit direct insert into auto-populated tables (#511)
 
 0.11.0 -- Oct 25, 2018
@@ -147,10 +167,10 @@
 * Bugfix in restriction of the form (A & B) * B (#463)
 * Improved error messages (#466)
 
-0.10.0 -- Jan 10, 2018 
+0.10.0 -- Jan 10, 2018
 ----------------------
 * Deletes are more efficient (#424)
-* ERD shows table definition on tooltip hover in Jupyter (#422) 
+* ERD shows table definition on tooltip hover in Jupyter (#422)
 * S3 external storage
 * Garbage collection for external sorage
 * Most operators and methods of tables can be invoked as class methods rather than instance methods (#407)
@@ -165,7 +185,7 @@
 * Implement union operator +
 * Implement file-based external storage
 
-0.8.0 -- Jul 26, 2017 
+0.8.0 -- Jul 26, 2017
 ---------------------
 Documentation and tutorials available at https://docs.datajoint.io and https://tutorials.datajoint.io
 * improved the ERD graphics and features using the graphviz libraries (#207, #333)
@@ -187,12 +207,12 @@ Documentation and tutorials available at https://docs.datajoint.io and https://t
 
 0.4.10 (#286) -- Feb 6, 2017
 ----------------------------
-* Removed Vagrant and Readthedocs support 
+* Removed Vagrant and Readthedocs support
 * Explicit saving of configuration (issue #284)
 
 0.4.9 (#285) -- Feb 2, 2017
 ---------------------------
-* Fixed setup.py for pip install 
+* Fixed setup.py for pip install
 
 0.4.7 (#281) -- Jan 24, 2017
 ----------------------------
@@ -229,11 +249,11 @@ Documentation and tutorials available at https://docs.datajoint.io and https://t
 0.3.8  -- Aug 2, 2016
 ---------------------
 * added the ``_update`` method in ``base_relation``. It allows updating values in existing tuples.
-* bugfix in reading values of type double.  Previously it was cast as float32. 
+* bugfix in reading values of type double.  Previously it was cast as float32.
 
 0.3.7  -- Jul 31, 2016
 ----------------------
-* added parameter ``ignore_extra_fields`` in ``insert`` 
+* added parameter ``ignore_extra_fields`` in ``insert``
 * ``insert(..., skip_duplicates=True)`` now relies on ``SELECT IGNORE``.  Previously it explicitly checked if tuple already exists.
 * table previews now include blob attributes displaying the string <BLOB>
 
@@ -259,7 +279,7 @@ Documentation and tutorials available at https://docs.datajoint.io and https://t
 * ERD() no longer text the context argument.
 * ERD.draw() now takes an optional context argument.  By default uses the caller's locals.
 
-0.3.2   
+0.3.2
 -----
 * Fixed issue #223:  ``insert`` can insert relations without fetching.
 * ERD() now takes the ``context`` argument, which specifies in which context to look for classes. The default is taken from the argument (schema or relation).
diff --git a/local-docker-compose.yml b/local-docker-compose.yml
index 4b12e74e0..19a8c7ae2 100644
--- a/local-docker-compose.yml
+++ b/local-docker-compose.yml
@@ -34,7 +34,7 @@ services:
       interval: 1s
   fakeservices.datajoint.io:
     <<: *net
-    image: datajoint/nginx:v0.0.16
+    image: datajoint/nginx:v0.0.19
     environment:
     - ADD_db_TYPE=DATABASE
     - ADD_db_ENDPOINT=db:3306
diff --git a/requirements.txt b/requirements.txt
index 363271d45..65c0c8b6f 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -10,3 +10,4 @@ minio>=7.0.0
 matplotlib
 cryptography
 otumat
+urllib3


Test patch:
diff --git a/test_requirements.txt b/test_requirements.txt
index 6f13c7c6d..0ed24a620 100644
--- a/test_requirements.txt
+++ b/test_requirements.txt
@@ -1,4 +1,4 @@
 nose
 nose-cov
 coveralls
-Faker
+faker
diff --git a/tests/schema.py b/tests/schema.py
index a0b336a1c..ccb5fcf31 100644
--- a/tests/schema.py
+++ b/tests/schema.py
@@ -439,3 +439,13 @@ class SessionDateA(dj.Lookup):
         ('mouse1', '2020-12-03'),
         ('mouse1', '2020-12-04')
     ]
+
+
+@schema
+class Stimulus(dj.Lookup):
+    definition = """
+    id: int
+    ---
+    contrast: int
+    brightness: int
+    """
diff --git a/tests/schema_simple.py b/tests/schema_simple.py
index c4ec45e00..9263fa01a 100644
--- a/tests/schema_simple.py
+++ b/tests/schema_simple.py
@@ -4,6 +4,10 @@
 import random
 import datajoint as dj
 import itertools
+import hashlib
+import uuid
+import faker
+
 
 from . import PREFIX, CONN_INFO
 import numpy as np
@@ -151,6 +155,55 @@ class DataB(dj.Lookup):
     contents = list(zip(range(5), range(5, 10)))
 
 
+@schema
+class Website(dj.Lookup):
+    definition = """
+    url_hash : uuid
+    ---
+    url : varchar(1000)
+    """
+
+    def insert1_url(self, url):
+        hashed = hashlib.sha1()
+        hashed.update(url.encode())
+        url_hash = uuid.UUID(bytes=hashed.digest()[:16])
+        self.insert1(dict(url=url, url_hash=url_hash), skip_duplicates=True)
+        return url_hash
+
+
+@schema
+class Profile(dj.Manual):
+    definition = """
+    ssn : char(11)
+    ---
+    name : varchar(70)
+    residence : varchar(255)
+    blood_group  : enum('A+', 'A-', 'AB+', 'AB-', 'B+', 'B-', 'O+', 'O-')
+    username : varchar(120)
+    birthdate : date
+    job : varchar(120)
+    sex : enum('M', 'F')
+    """
+
+    class Website(dj.Part):
+        definition = """
+        -> master
+        -> Website
+        """
+
+    def populate_random(self, n=10):
+        faker.Faker.seed(0)
+        fake = faker.Faker()
+        faker.Faker.seed(0)  # make tests deterministic
+        for _ in range(n):
+            profile = fake.profile()
+            with self.connection.transaction:
+                self.insert1(profile, ignore_extra_fields=True)
+                for url in profile['website']:
+                    self.Website().insert1(
+                        dict(ssn=profile['ssn'], url_hash=Website().insert1_url(url)))
+
+
 @schema
 class TTestUpdate(dj.Lookup):
     definition = """
diff --git a/tests/test_aggr_regressions.py b/tests/test_aggr_regressions.py
index 1cadfdf34..7947e6f18 100644
--- a/tests/test_aggr_regressions.py
+++ b/tests/test_aggr_regressions.py
@@ -3,10 +3,11 @@
 """
 
 import itertools
-from nose.tools import assert_equal, raises
+from nose.tools import assert_equal
 import datajoint as dj
 from . import PREFIX, CONN_INFO
-
+import uuid
+from .schema_uuid import Topic, Item, top_level_namespace_id
 schema = dj.Schema(PREFIX + '_aggr_regress', connection=dj.conn(**CONN_INFO))
 
 # --------------- ISSUE 386 -------------------
@@ -103,3 +104,27 @@ def test_issue558_part2():
     d = dict(id=3, id2=5)
     assert_equal(len(X & d), len((X & d).proj(id2='3')))
 
+
+def test_left_join_len():
+    Topic().add('jeff')
+    Item.populate()
+    Topic().add('jeff2')
+    Topic().add('jeff3')
+    q = Topic.join(Item - dict(topic_id=uuid.uuid5(top_level_namespace_id, 'jeff')),
+                   left=True)
+    qf = q.fetch()
+    assert len(q) == len(qf)
+
+
+def test_union_join():
+    # https://github.com/datajoint/datajoint-python/issues/930
+    A.insert(zip([100, 200, 300, 400, 500, 600]))
+    B.insert([(100, 11), (200, 22), (300, 33), (400, 44)])
+    q1 = B & 'id < 300'
+    q2 = B & 'id > 300'
+
+    expected_data = [{'id': 0, 'id2': 5}, {'id': 1, 'id2': 6}, {'id': 2, 'id2': 7},
+                     {'id': 3, 'id2': 8}, {'id': 4, 'id2': 9}, {'id': 100, 'id2': 11},
+                     {'id': 200, 'id2': 22}, {'id': 400, 'id2': 44}]
+
+    assert ((q1 + q2) * A).fetch(as_dict=True) == expected_data
diff --git a/tests/test_autopopulate.py b/tests/test_autopopulate.py
index c9cb7f97d..1875a6743 100644
--- a/tests/test_autopopulate.py
+++ b/tests/test_autopopulate.py
@@ -1,13 +1,13 @@
 from nose.tools import assert_equal, assert_false, assert_true, raises
-from . import schema
+from . import schema, PREFIX
 from datajoint import DataJointError
+import datajoint as dj
 
 
 class TestPopulate:
     """
     Test base relations: insert, delete
     """
-
     def setUp(self):
         self.user = schema.User()
         self.subject = schema.Subject()
@@ -52,7 +52,7 @@ def test_populate(self):
 
     def test_allow_direct_insert(self):
         assert_true(self.subject, 'root tables are empty')
-        key = self.subject.fetch('KEY')[0]
+        key = self.subject.fetch('KEY', limit=1)[0]
         key['experiment_id'] = 1000
         key['experiment_date'] = '2018-10-30'
         self.experiment.insert1(key, allow_direct_insert=True)
@@ -64,3 +64,39 @@ def test_allow_insert(self):
         key['experiment_id'] = 1001
         key['experiment_date'] = '2018-10-30'
         self.experiment.insert1(key)
+
+    def test_load_dependencies(self):
+        schema = dj.Schema(f'{PREFIX}_load_dependencies_populate')
+
+        @schema
+        class ImageSource(dj.Lookup):
+            definition = """
+            image_source_id: int
+            """
+            contents = [(0,)]
+
+        @schema
+        class Image(dj.Imported):
+            definition = """
+            -> ImageSource
+            ---
+            image_data: longblob
+            """
+
+            def make(self, key):
+                self.insert1(dict(key, image_data=dict()))
+
+        Image.populate()
+
+        @schema
+        class Crop(dj.Computed):
+            definition = """
+            -> Image
+            ---
+            crop_image: longblob
+            """
+
+            def make(self, key):
+                self.insert1(dict(key, crop_image=dict()))
+
+        Crop.populate()
diff --git a/tests/test_blob.py b/tests/test_blob.py
index 225fb775c..61e60a8d0 100644
--- a/tests/test_blob.py
+++ b/tests/test_blob.py
@@ -23,6 +23,9 @@ def test_pack():
     x = np.random.randn(10)
     assert_array_equal(x, unpack(pack(x)), "Arrays do not match!")
 
+    x = 7j
+    assert_equal(x, unpack(pack(x)), "Complex scalar does not match")
+
     x = np.float32(np.random.randn(3, 4, 5))
     assert_array_equal(x, unpack(pack(x)), "Arrays do not match!")
 
diff --git a/tests/test_cascading_delete.py b/tests/test_cascading_delete.py
index 6988d432a..6e730590a 100644
--- a/tests/test_cascading_delete.py
+++ b/tests/test_cascading_delete.py
@@ -1,6 +1,6 @@
-from nose.tools import assert_false, assert_true, assert_equal
+from nose.tools import assert_false, assert_true, assert_equal, raises
 import datajoint as dj
-from .schema_simple import A, B, D, E, L
+from .schema_simple import A, B, D, E, L, Website, Profile
 from .schema import ComplexChild, ComplexParent
 
 
@@ -27,17 +27,19 @@ def test_delete_tree():
 
     @staticmethod
     def test_stepwise_delete():
-        assert_false(dj.config['safemode'], 'safemode must be off for testing') #TODO: just turn it off instead of warning
-        assert_true(L() and A() and B() and B.C(), 'schema population failed as a precondition to test')
+        assert not dj.config['safemode'], 'safemode must be off for testing'
+        assert L() and A() and B() and B.C(), 'schema population failed'
         B.C().delete(force=True)
-        assert_false(B.C(), 'failed to delete child tables')
+        assert not B.C(), 'failed to delete child tables'
         B().delete()
-        assert_false(B(), 'failed to delete the parent table following child table deletion')
+        assert not B(), \
+            'failed to delete from the parent table following child table deletion'
 
     @staticmethod
     def test_delete_tree_restricted():
-        assert_false(dj.config['safemode'], 'safemode must be off for testing')
-        assert_true(L() and A() and B() and B.C() and D() and E() and E.F(), 'schema is not populated')
+        assert not dj.config['safemode'], 'safemode must be off for testing'
+        assert L() and A() and B() and B.C() and D() and E() and E.F(), \
+            'schema is not populated'
         cond = 'cond_in_a'
         rel = A() & cond
         rest = dict(
@@ -48,19 +50,14 @@ def test_delete_tree_restricted():
             E=len(E()-rel),
             F=len(E.F()-rel))
         rel.delete()
-        assert_false(rel or
-                     (B() & rel) or
-                     (B.C() & rel) or
-                     (D() & rel) or
-                     (E() & rel) or
-                     (E.F() & rel),
-                     'incomplete delete')
-        assert_equal(len(A()), rest['A'], 'invalid delete restriction')
-        assert_equal(len(B()), rest['B'], 'invalid delete restriction')
-        assert_equal(len(B.C()), rest['C'], 'invalid delete restriction')
-        assert_equal(len(D()), rest['D'], 'invalid delete restriction')
-        assert_equal(len(E()), rest['E'], 'invalid delete restriction')
-        assert_equal(len(E.F()), rest['F'], 'invalid delete restriction')
+        assert not (rel or B() & rel or B.C() & rel or D() & rel or E() & rel
+                    or (E.F() & rel)), 'incomplete delete'
+        assert len(A()) == rest['A'], 'invalid delete restriction'
+        assert len(B()) == rest['B'], 'invalid delete restriction'
+        assert len(B.C()) == rest['C'], 'invalid delete restriction'
+        assert len(D()) == rest['D'], 'invalid delete restriction'
+        assert len(E()) == rest['E'], 'invalid delete restriction'
+        assert len(E.F()) == rest['F'], 'invalid delete restriction'
 
     @staticmethod
     def test_delete_lookup():
@@ -96,3 +93,18 @@ def test_delete_complex_keys():
         (ComplexParent & restriction).delete()
         assert len(ComplexParent & restriction) == 0, 'Parent record was not deleted'
         assert len(ComplexChild & restriction) == 0, 'Child record was not deleted'
+
+    def test_delete_master(self):
+        Profile().populate_random()
+        Profile().delete()
+
+    @raises(dj.DataJointError)
+    def test_delete_parts(self):
+        """test issue #151"""
+        Profile().populate_random()
+        Website().delete()
+
+    @raises(dj.DataJointError)
+    def test_drop_part(self):
+        """test issue #374"""
+        Website().drop()
diff --git a/tests/test_declare.py b/tests/test_declare.py
index cce2d6366..f0128e2b5 100644
--- a/tests/test_declare.py
+++ b/tests/test_declare.py
@@ -160,6 +160,37 @@ def test_dependencies():
         assert_set_equal(set(s.full_table_name for s in channel.parents(primary=True, as_objects=True)),
                          {ephys.full_table_name})
 
+    @staticmethod
+    def test_descendants_only_contain_part_table():
+        """issue #927"""
+
+        @schema
+        class A(dj.Manual):
+            definition = """
+            a: int
+            """
+
+        @schema
+        class B(dj.Manual):
+            definition = """
+            -> A
+            b: int
+            """
+
+        @schema
+        class Master(dj.Manual):
+            definition = """
+            table_master: int
+            """
+
+            class Part(dj.Part):
+                definition = """
+                -> master
+                -> B
+                """
+
+        assert A.descendants() == ['`djtest_test1`.`a`', '`djtest_test1`.`b`', '`djtest_test1`.`master__part`']
+
     @staticmethod
     @raises(dj.DataJointError)
     def test_bad_attribute_name():
diff --git a/tests/test_dependencies.py b/tests/test_dependencies.py
index 2b7687bb5..de9b13603 100644
--- a/tests/test_dependencies.py
+++ b/tests/test_dependencies.py
@@ -9,13 +9,17 @@ def test_unite_master_parts():
         ['`s`.`a`', '`s`.`a__q`', '`s`.`a__r`', '`s`.`b`', '`s`.`b__q`', '`s`.`c`', '`s`.`c__q`', '`s`.`d`'])
     assert_list_equal(unite_master_parts(
         [
+            '`lab`.`#equipment`',
             '`cells`.`cell_analysis_method`',
             '`cells`.`cell_analysis_method_task_type`',
             '`cells`.`cell_analysis_method_users`',
             '`cells`.`favorite_selection`',
             '`cells`.`cell_analysis_method__cell_selection_params`',
+            '`lab`.`#equipment__config`',
             '`cells`.`cell_analysis_method__field_detect_params`']),
         [
+            '`lab`.`#equipment`',
+            '`lab`.`#equipment__config`',
             '`cells`.`cell_analysis_method`',
             '`cells`.`cell_analysis_method__cell_selection_params`',
             '`cells`.`cell_analysis_method__field_detect_params`',
diff --git a/tests/test_erd.py b/tests/test_erd.py
index 6c4ae24b7..d1905e418 100644
--- a/tests/test_erd.py
+++ b/tests/test_erd.py
@@ -26,6 +26,7 @@ def test_decorator():
     @staticmethod
     def test_dependencies():
         deps = schema.connection.dependencies
+        deps.load()
         assert_true(all(cls.full_table_name in deps for cls in (A, B, B.C, D, E, E.F, L)))
         assert_true(set(A().children()) == set([B.full_table_name, D.full_table_name]))
         assert_true(set(D().parents(primary=True)) == set([A.full_table_name]))
diff --git a/tests/test_external.py b/tests/test_external.py
index 22fee51ab..fb85a2afc 100644
--- a/tests/test_external.py
+++ b/tests/test_external.py
@@ -3,11 +3,10 @@
 from nose.tools import assert_true, assert_equal
 from datajoint.external import ExternalTable
 from datajoint.blob import pack, unpack
-
-from .schema_external import schema
 import datajoint as dj
-from .schema_external import stores_config
-from .schema_external import SimpleRemote
+from .schema_external import stores_config, SimpleRemote, Simple, schema
+import os
+
 current_location_s3 = dj.config['stores']['share']['location']
 current_location_local = dj.config['stores']['local']['location']
 
@@ -47,6 +46,9 @@ def test_s3_leading_slash(index=100, store='share'):
     """
     s3 external storage configured with leading slash
     """
+
+    oldConfig = dj.config['stores'][store]['location']
+
     value = np.array([1, 2, 3])
 
     id = index
@@ -97,9 +99,28 @@ def test_s3_leading_slash(index=100, store='share'):
     assert_true(np.array_equal(
         value, (SimpleRemote & 'simple={}'.format(id)).fetch1('item')))
 
+    dj.config['stores'][store]['location'] = oldConfig
+
 
 def test_file_leading_slash():
     """
     file external storage configured with leading slash
     """
     test_s3_leading_slash(index=200, store='local')
+
+
+def test_remove_fail():
+    # https://github.com/datajoint/datajoint-python/issues/953
+    data = dict(simple=2, item=[1, 2, 3])
+    Simple.insert1(data)
+    path1 = dj.config['stores']['local']['location'] + '/djtest_extern/4/c/'
+    currentMode = int(oct(os.stat(path1).st_mode), 8)
+    os.chmod(path1, 0o40555)
+    (Simple & 'simple=2').delete()
+    listOfErrors = schema.external['local'].delete(delete_external_files=True)
+    assert len(listOfErrors) == 1, 'unexpected number of errors'
+    assert len(schema.external['local'] & dict(hash=listOfErrors[0][0])) == 1, \
+        'unexpected number of rows in external table'
+    # ---------------------CLEAN UP--------------------
+    os.chmod(path1, currentMode)
+    listOfErrors = schema.external['local'].delete(delete_external_files=True)
\ No newline at end of file
diff --git a/tests/test_fetch.py b/tests/test_fetch.py
index cb1ba3a4f..e074ee7ca 100644
--- a/tests/test_fetch.py
+++ b/tests/test_fetch.py
@@ -6,6 +6,7 @@
 import pandas
 import warnings
 from . import schema
+from .schema import Parent, Stimulus
 import datajoint as dj
 import os
 
@@ -287,3 +288,34 @@ def test_query_caching(self):
 
         # reset cache directory state (will fail if purge was unsuccessful)
         os.rmdir(os.path.expanduser('~/dj_query_cache'))
+
+    def test_fetch_group_by(self):
+        # https://github.com/datajoint/datajoint-python/issues/914
+
+        assert Parent().fetch('KEY', order_by='name') == [{'parent_id': 1}]
+
+    def test_dj_u_distinct(self):
+        # Test developed to see if removing DISTINCT from the select statement
+        # generation breakes the dj.U universal set imlementation
+
+        # Contents to be inserted
+        contents = [
+            (1,2,3),
+            (2,2,3),
+            (3,3,2),
+            (4,5,5)
+        ]
+        Stimulus.insert(contents)
+
+        # Query the whole table
+        test_query = Stimulus()
+
+        # Use dj.U to create a list of unique contrast and brightness combinations
+        result = dj.U('contrast', 'brightness') & test_query
+        expected_result = [{'contrast': 2, 'brightness': 3},
+                           {'contrast': 3, 'brightness': 2},
+                           {'contrast': 5, 'brightness': 5}]
+
+        fetched_result = result.fetch(as_dict=True, order_by=('contrast', 'brightness'))
+        Stimulus.delete_quick()
+        assert fetched_result == expected_result
diff --git a/tests/test_groupby.py b/tests/test_groupby.py
new file mode 100644
index 000000000..cdf8d7b53
--- /dev/null
+++ b/tests/test_groupby.py
@@ -0,0 +1,11 @@
+from .schema_simple import A, D
+
+
+def test_aggr_with_proj():
+    # issue #944 - only breaks with MariaDB
+    # MariaDB implements the SQL:1992 standard that prohibits fields in the select statement that are 
+    # not also in the GROUP BY statement.
+    # An improved specification in SQL:2003 allows fields that are functionally dependent on the group by 
+    # attributes to be allowed in the select. This behavior is implemented by MySQL correctly but not MariaDB yet. 
+    # See MariaDB issue: https://jira.mariadb.org/browse/MDEV-11588
+    A.aggr(D.proj(m='id_l'), ..., n='max(m) - min(m)')
diff --git a/tests/test_jobs.py b/tests/test_jobs.py
index 5990c3019..65864879e 100644
--- a/tests/test_jobs.py
+++ b/tests/test_jobs.py
@@ -122,3 +122,23 @@ def test_long_error_message():
     assert_true(error_message == short_error_message, 'error messages do not agree')
     assert_false(error_message.endswith(TRUNCATION_APPENDIX), 'error message should not be truncated')
     schema.schema.jobs.delete()
+
+
+def test_long_error_stack():
+    # clear out jobs table
+    schema.schema.jobs.delete()
+
+    # create long error stack
+    STACK_SIZE = 89942  # Does not fit into small blob (should be 64k, but found to be higher)
+    long_error_stack = ''.join(random.choice(string.ascii_letters) for _ in range(STACK_SIZE))
+    assert subjects
+    table_name = 'fake_table'
+
+    key = subjects.fetch('KEY')[0]
+
+    # test long error stack
+    schema.schema.jobs.reserve(table_name, key)
+    schema.schema.jobs.error(table_name, key, 'error message', long_error_stack)
+    error_stack = schema.schema.jobs.fetch1('error_stack')
+    assert error_stack == long_error_stack, 'error stacks do not agree'
+    schema.schema.jobs.delete()
diff --git a/tests/test_relational_operand.py b/tests/test_relational_operand.py
index 108bf895b..e558650a2 100644
--- a/tests/test_relational_operand.py
+++ b/tests/test_relational_operand.py
@@ -13,6 +13,8 @@
 from .schema import (Experiment, TTest3, Trial, Ephys, Child, Parent, SubjectA, SessionA,
                      SessionStatusA, SessionDateA)
 
+from . import PREFIX, CONN_INFO
+
 
 def setup():
     """
@@ -177,6 +179,15 @@ def test_project():
         assert_equal(len((D() & cond).proj()), len((D() & cond)),
                      'projection failed: altered its argument''s cardinality')
 
+    @staticmethod
+    def test_rename_non_dj_attribute():
+        schema = PREFIX + '_test1'
+        connection = dj.conn(**CONN_INFO)
+        connection.query(f'CREATE TABLE {schema}.test_table (oldID int PRIMARY KEY)').fetchall()
+        mySchema = dj.VirtualModule(schema, schema)
+        assert 'oldID' not in  mySchema.TestTable.proj(new_name='oldID').heading.attributes.keys(), 'Failed to rename attribute correctly'
+        connection.query(f'DROP TABLE {schema}.test_table')
+
     @staticmethod
     def test_union():
         x = set(zip(*IJ.fetch('i', 'j')))
@@ -505,3 +516,13 @@ def test_joins_with_aggregation():
         session_dates = ((SessionDateA * (subj_query & 'date_trained<"2020-12-21"')) &
                          'session_date<date_trained')
         assert len(session_dates) == 1
+
+    @staticmethod
+    def test_union_multiple():
+        # https://github.com/datajoint/datajoint-python/issues/926
+        q1 = IJ & dict(j=2)
+        q2 = (IJ & dict(j=2, i=0)) + (IJ & dict(j=2, i=1)) + (IJ & dict(j=2, i=2))
+        x = set(zip(*q1.fetch('i', 'j')))
+        y = set(zip(*q2.fetch('i', 'j')))
+        assert x == y
+        assert q1.fetch(as_dict=True) == q2.fetch(as_dict=True)
diff --git a/tests/test_s3.py b/tests/test_s3.py
index 73ae89f9e..288a05f05 100644
--- a/tests/test_s3.py
+++ b/tests/test_s3.py
@@ -1,9 +1,12 @@
 from . import S3_CONN_INFO
-
 from minio import Minio
 import urllib3
 import certifi
-from nose.tools import assert_true
+from nose.tools import assert_true, raises
+from .schema_external import schema, SimpleRemote
+from datajoint.errors import DataJointError
+from datajoint.hash import uuid_from_buffer
+from datajoint.blob import pack
 
 
 class TestS3:
@@ -51,3 +54,35 @@ def test_connection_secure():
             http_client=http_client)
 
         assert_true(minio_client.bucket_exists(S3_CONN_INFO['bucket']))
+
+    @staticmethod
+    @raises(DataJointError)
+    def test_remove_object_exception():
+        # https://github.com/datajoint/datajoint-python/issues/952
+
+        # Insert some test data and remove it so that the external table is populated
+        test = [1, [1, 2, 3]]
+        SimpleRemote.insert1(test)
+        SimpleRemote.delete()
+
+        # Save the old external table minio client
+        old_client = schema.external['share'].s3.client
+
+        # Apply our new minio client which has a user that does not exist
+        schema.external['share'].s3.client = Minio(
+            S3_CONN_INFO['endpoint'],
+            access_key='jeffjeff',
+            secret_key='jeffjeff',
+            secure=False)
+
+        # This method returns a list of errors
+        error_list = schema.external['share'].delete(delete_external_files=True,
+                                                     errors_as_string=False)
+
+        # Teardown
+        schema.external['share'].s3.client = old_client
+        schema.external['share'].delete(delete_external_files=True)
+
+        # Raise the error we want if the error matches the expected uuid
+        if str(error_list[0][0]) == str(uuid_from_buffer(pack(test[1]))):
+            raise error_list[0][2]
diff --git a/tests/test_schema.py b/tests/test_schema.py
index 42d4e0c0e..1ff41efdf 100644
--- a/tests/test_schema.py
+++ b/tests/test_schema.py
@@ -124,7 +124,8 @@ def test_list_tables():
     # https://github.com/datajoint/datajoint-python/issues/838
     assert(set(['reserved_word', '#l', '#a', '__d', '__b', '__b__c', '__e', '__e__f',
                 '#outfit_launch', '#outfit_launch__outfit_piece', '#i_j', '#j_i',
-                '#t_test_update', '#data_a', '#data_b', 'f', '#argmax_test'
+                '#t_test_update', '#data_a', '#data_b', 'f', '#argmax_test',
+                '#website', 'profile', 'profile__website'
                 ]) == set(schema_simple.list_tables()))
 
 
diff --git a/tests/test_schema_keywords.py b/tests/test_schema_keywords.py
index 577524511..50cb25558 100644
--- a/tests/test_schema_keywords.py
+++ b/tests/test_schema_keywords.py
@@ -3,7 +3,7 @@
 from nose.tools import assert_true
 
 
-schema = dj.Schema(PREFIX + '_keywords', locals(), connection=dj.conn(**CONN_INFO))
+schema = dj.Schema(PREFIX + '_keywords', connection=dj.conn(**CONN_INFO))
 
 
 @schema