Repository: napari/napari

Issue:
Setting transforms on existing layer not pure
## 🐛 Bug

I'm not sure if this is by design, or a bug... but setting `layer.shear` with a full shear matrix is impure: calling it with the same value repeatedly gives you different results.

## To Reproduce

```python
import napari
import numpy as np

viewer = napari.Viewer()
layer = viewer.add_image(np.random.rand(64,64,64))

mat = np.eye(3)
mat[2,0] = 0.5
layer.shear = mat
layer.shear  # array([-0. ,  0.4,  0. ])
layer.shear = mat
layer.shear  # array([-0.        ,  0.27586207,  0.        ])
layer.shear = mat
layer.shear  # array([-0.        ,  0.21253985,  0.        ])
```

yes, warnings are provided when I set those, but it's not clear from the warnings (until you look at the source code for the shear setter showing the decomposition and recomposition of the full matrix) why setting shear with the same value yields a different result.

```ipytb
UserWarning: Non upper diagonal shear matrix passed so reseting rotate to the identity.
UserWarning: Non-orthogonal slicing is being requested, but is not fully supported. Data is displayed without applying an out-of-slice rotation or shear component.
```


<!-- If you have a code sample, error messages, stack traces, please provide it here as well -->

## Expected behavior

setting a layer attribute should not depend on previous state.  If the intention was to _apply_ shear to whatever the current matrix is, then I would have expected it to be a method (like `apply_shear`) rather setting an attribute `layer.shear = ...`

Affine transform upper triangular shear matrix values are looking wrong
## 🐛 Bug

Looks like the `napari.utils.transforms.Affine`  `shear`  with the upper triangular matrix is not working as expected.


## To Reproduce
```
ut_shear_affine = Affine(shear=[0, np.tan(np.deg2rad(90 - 31.8)), 0])

test_shear = shear_matrix_from_angle(31.8)
shear_affine_with_func = Affine(shear=test_shear)

print("shear with upper trangular matrix")
print(ut_shear_affine .affine_matrix.T)

print("using shear_matrix_from_angle function ")
print(shear_affine_with_func.affine_matrix.T)
```


output:
```
shear with upper trangular matrix
[[1.         0.         0.         0.        ]
 [0.         1.         0.         0.        ]
 [1.61283489 0.         1.         0.        ]
 [0.         0.         0.         1.        ]]

using shear_matrix_from_angle function
[[1.         0.         1.61283489 0.        ]
 [0.         1.         0.         0.        ]
 [0.         0.         1.         0.        ]
 [0.         0.         0.         1.        ]]
```

## Expected behavior/output

```
shear with upper trangular matrix
[[1.         0.         1.61283489 0.        ]
 [0.         1.         0.         0.        ]
 [0.         0.         1.         0.        ]
 [0.         0.         0.         1.        ]]

using shear_matrix_from_angle function
[[1.         0.         1.61283489 0.        ]
 [0.         1.         0.         0.        ]
 [0.         0.         1.         0.        ]
 [0.         0.         0.         1.        ]]
```

## Environment

```
napari: 0.4.1.dev15+gbc3298b
Platform: Windows-10-10.0.19041-SP0
Python: 3.8.1 (default, Mar 2 2020, 13:06:26) [MSC v.1916 64 bit (AMD64)]
Qt: 5.12.9
PySide2: 5.13.2
NumPy: 1.18.1
SciPy: 1.5.3
Dask: 2.30.0
VisPy: 0.6.5

GL version: 4.6.0 NVIDIA 457.09
MAX_TEXTURE_SIZE: 32768

Plugins:
- napari_plugin_engine: 0.1.8
- svg: 0.1.4
```




Discussion:
> I'm not sure if this is by design, or a bug... but setting layer.shear with a full shear matrix is impure: calling it with the same value repeatedly gives you different results.

Yeah, agreed we should fix. Maybe rather than emit "UserWarning: Non upper diagonal shear matrix passed so reseting rotate to the identity." I should have just thrown an error but I didn't actually realize it would be this bad with that example (I would have thought it would be stable, but you'd loose any additional rotation information each time), so maybe more is going wrong that I thought.

The issue is that our shear decomposition always goes to an upper triangular matrix, but you're passing in a full shear matrix that is lower triangular, so something's got to give, but it should be stable.
> I should have just thrown an error but I didn't actually realize it would be this bad with that example ... The issue is that our shear decomposition always goes to an upper triangular matrix, but you're passing in a full shear matrix that is lower triangular

fwiw, I noticed this first with `shear_matrix_from_angle` ... since that also creates a lower triangular matrix.  so setting `layer.shear = shear_matrix_from_angle(31)` has the same outcome
Just did some more looking into this, one fix would be to add `scale = np.ones(len(scale))` here https://github.com/napari/napari/blob/master/napari/utils/transforms/transforms.py#L378-L383

This would then fix the above issue and setting shear would be stateless. We could do this as a quick fix for 
`layer.shear = shear_matrix_from_angle(31)` but there's a deeper issue that our reduced representation is upper triangular and so we're not really able to have a nice round trip with a lower triangular input :-( Very curious what you'd recommend here, I'd love to see this fixed
If user passes a full matrix to shear we should detect if lower triangular/ upper triangular and decompose as appropriate. This would prevent the above bug. We can keep track of this and return the decomposition as appropriate.

We should also add a `homogeneous` keyword to `shear_matrix_from_angle`.

Thanks @jni and @tlambert03 for help figuring this out!!!!
Yes, I think what's happening is `shear_matrix_from_angle` is producing lower triangular by default, but Affine constructs upper triangular by default. While not identical to #1826 this is not unrelated, and we should make sure both of these meet user expectations and needs by the next release.

Resolving patch:
diff --git a/napari/utils/transforms/transform_utils.py b/napari/utils/transforms/transform_utils.py
--- a/napari/utils/transforms/transform_utils.py
+++ b/napari/utils/transforms/transform_utils.py
@@ -1,4 +1,5 @@
 import numpy as np
+import scipy.linalg
 
 
 def compose_linear_matrix(rotate, scale, shear) -> np.array:
@@ -149,7 +150,9 @@ def embed_in_identity_matrix(matrix, ndim):
         return full_matrix
 
 
-def decompose_linear_matrix(matrix) -> (np.array, np.array, np.array):
+def decompose_linear_matrix(
+    matrix, upper_triangular=True
+) -> (np.array, np.array, np.array):
     """Decompose linear transform matrix into rotate, scale, shear.
 
     Decomposition is based on code from https://github.com/matthew-brett/transforms3d.
@@ -160,6 +163,9 @@ def decompose_linear_matrix(matrix) -> (np.array, np.array, np.array):
     ----------
     matrix : np.array shape (N, N)
         nD array representing the composed linear transform.
+    upper_triangular : bool
+        Whether to decompose shear into an upper triangular or
+        lower triangular matrix.
 
     Returns
     -------
@@ -174,22 +180,31 @@ def decompose_linear_matrix(matrix) -> (np.array, np.array, np.array):
         in leading dimensions, so that, for example, a scale of [4, 18, 34] in
         3D can be used as a scale of [1, 4, 18, 34] in 4D without modification.
         An empty translation vector implies no scaling.
-    shear : n-D array
-        An n-D shear matrix.
+    shear : 1-D array or n-D array
+        1-D array of upper triangular values or an n-D matrix if lower
+        triangular.
     """
     n = matrix.shape[0]
 
-    rotate, upper_tri = np.linalg.qr(matrix)
+    if upper_triangular:
+        rotate, tri = scipy.linalg.qr(matrix)
+    else:
+        upper_tri, rotate = scipy.linalg.rq(matrix.T)
+        rotate = rotate.T
+        tri = upper_tri.T
 
-    scale = np.diag(upper_tri).copy()
-    upper_tri_normalized = upper_tri / scale[:, np.newaxis]
+    scale = np.diag(tri).copy()
+    tri_normalized = tri / scale[:, np.newaxis]
 
     if np.linalg.det(rotate) < 0:
         scale[0] *= -1
-        upper_tri[0] *= -1
-        rotate = matrix @ np.linalg.inv(upper_tri)
+        tri[0] *= -1
+        rotate = matrix @ np.linalg.inv(tri)
 
-    shear = upper_tri_normalized[np.triu(np.ones((n, n)), 1).astype(bool)]
+    if upper_triangular:
+        shear = tri_normalized[np.triu(np.ones((n, n)), 1).astype(bool)]
+    else:
+        shear = tri_normalized
 
     return rotate, scale, shear
 
@@ -215,3 +230,53 @@ def shear_matrix_from_angle(angle, ndim=3, axes=(-1, 0)):
     matrix = np.eye(ndim)
     matrix[axes] = np.tan(np.deg2rad(90 - angle))
     return matrix
+
+
+def is_matrix_upper_triangular(matrix):
+    """Check if a matrix is upper triangular.
+
+    Parameters
+    ----------
+    matrix : np.ndarray
+        Matrix to be checked.
+
+    Returns
+    -------
+    bool
+        Whether matrix is upper triangular or not.
+    """
+    return np.allclose(matrix, np.triu(matrix))
+
+
+def is_matrix_lower_triangular(matrix):
+    """Check if a matrix is lower triangular.
+
+    Parameters
+    ----------
+    matrix : np.ndarray
+        Matrix to be checked.
+
+    Returns
+    -------
+    bool
+        Whether matrix is lower triangular or not.
+    """
+    return np.allclose(matrix, np.tril(matrix))
+
+
+def is_matrix_triangular(matrix):
+    """Check if a matrix is triangular.
+
+    Parameters
+    ----------
+    matrix : np.ndarray
+        Matrix to be checked.
+
+    Returns
+    -------
+    bool
+        Whether matrix is triangular or not.
+    """
+    return is_matrix_upper_triangular(matrix) or is_matrix_lower_triangular(
+        matrix
+    )
diff --git a/napari/utils/transforms/transforms.py b/napari/utils/transforms/transforms.py
--- a/napari/utils/transforms/transforms.py
+++ b/napari/utils/transforms/transforms.py
@@ -1,4 +1,3 @@
-import warnings
 from typing import Sequence
 
 import numpy as np
@@ -9,6 +8,8 @@
     compose_linear_matrix,
     decompose_linear_matrix,
     embed_in_identity_matrix,
+    is_matrix_triangular,
+    is_matrix_upper_triangular,
 )
 
 
@@ -306,6 +307,7 @@ def __init__(
         name=None,
     ):
         super().__init__(name=name)
+        self._upper_triangular = True
 
         if affine_matrix is not None:
             linear_matrix = affine_matrix[:-1, :-1]
@@ -317,6 +319,18 @@ def __init__(
                 rotate = np.eye(len(scale))
             if shear is None:
                 shear = np.eye(len(scale))
+            else:
+                if np.array(shear).ndim == 2:
+                    if is_matrix_triangular(shear):
+                        self._upper_triangular = is_matrix_upper_triangular(
+                            shear
+                        )
+                    else:
+                        raise ValueError(
+                            f'Only upper triangular or lower triangular matrices are '
+                            f'accepted for shear, got {shear}. For other matrices, set the '
+                            f'affine_matrix or linear_matrix directly.'
+                        )
             linear_matrix = compose_linear_matrix(rotate, scale, shear)
 
         ndim = max(linear_matrix.shape[0], len(translate))
@@ -347,40 +361,57 @@ def ndim(self) -> int:
     @property
     def scale(self) -> np.array:
         """Return the scale of the transform."""
-        return decompose_linear_matrix(self.linear_matrix)[1]
+        return decompose_linear_matrix(
+            self.linear_matrix, upper_triangular=self._upper_triangular
+        )[1]
 
     @scale.setter
     def scale(self, scale):
         """Set the scale of the transform."""
-        rotate, _, shear = decompose_linear_matrix(self.linear_matrix)
+        rotate, _, shear = decompose_linear_matrix(
+            self.linear_matrix, upper_triangular=self._upper_triangular
+        )
         self.linear_matrix = compose_linear_matrix(rotate, scale, shear)
 
     @property
     def rotate(self) -> np.array:
         """Return the rotation of the transform."""
-        return decompose_linear_matrix(self.linear_matrix)[0]
+        return decompose_linear_matrix(
+            self.linear_matrix, upper_triangular=self._upper_triangular
+        )[0]
 
     @rotate.setter
     def rotate(self, rotate):
         """Set the rotation of the transform."""
-        _, scale, shear = decompose_linear_matrix(self.linear_matrix)
+        _, scale, shear = decompose_linear_matrix(
+            self.linear_matrix, upper_triangular=self._upper_triangular
+        )
         self.linear_matrix = compose_linear_matrix(rotate, scale, shear)
 
     @property
     def shear(self) -> np.array:
         """Return the shear of the transform."""
-        return decompose_linear_matrix(self.linear_matrix)[2]
+        return decompose_linear_matrix(
+            self.linear_matrix, upper_triangular=self._upper_triangular
+        )[2]
 
     @shear.setter
     def shear(self, shear):
         """Set the shear of the transform."""
-        rotate, scale, _ = decompose_linear_matrix(self.linear_matrix)
         if np.array(shear).ndim == 2:
-            warnings.warn(
-                'Non upper diagonal shear matrix passed so '
-                'reseting rotate to the identity.'
-            )
-            rotate = np.eye(rotate.shape[0])
+            if is_matrix_triangular(shear):
+                self._upper_triangular = is_matrix_upper_triangular(shear)
+            else:
+                raise ValueError(
+                    f'Only upper triangular or lower triangular matrices are '
+                    f'accepted for shear, got {shear}. For other matrices, set the '
+                    f'affine_matrix or linear_matrix directly.'
+                )
+        else:
+            self._upper_triangular = True
+        rotate, scale, _ = decompose_linear_matrix(
+            self.linear_matrix, upper_triangular=self._upper_triangular
+        )
         self.linear_matrix = compose_linear_matrix(rotate, scale, shear)
 
     @property


Test patch:
diff --git a/napari/utils/transforms/_tests/test_transform_utils.py b/napari/utils/transforms/_tests/test_transform_utils.py
--- a/napari/utils/transforms/_tests/test_transform_utils.py
+++ b/napari/utils/transforms/_tests/test_transform_utils.py
@@ -1,26 +1,35 @@
 import numpy as np
+import pytest
 
 from napari.utils.transforms.transform_utils import (
     compose_linear_matrix,
     decompose_linear_matrix,
+    is_matrix_lower_triangular,
+    is_matrix_triangular,
+    is_matrix_upper_triangular,
     shear_matrix_from_angle,
 )
 
 
-def test_decompose_linear_matrix():
+@pytest.mark.parametrize('upper_triangular', [True, False])
+def test_decompose_linear_matrix(upper_triangular):
     """Test composing and decomposing a linear matrix."""
     np.random.seed(0)
 
     # Decompose linear matrix
     A = np.random.random((2, 2))
-    rotate, scale, shear = decompose_linear_matrix(A)
+    rotate, scale, shear = decompose_linear_matrix(
+        A, upper_triangular=upper_triangular
+    )
 
     # Compose linear matrix and check it matches
     B = compose_linear_matrix(rotate, scale, shear)
     np.testing.assert_almost_equal(A, B)
 
     # Decompose linear matrix and check it matches
-    rotate_B, scale_B, shear_B = decompose_linear_matrix(B)
+    rotate_B, scale_B, shear_B = decompose_linear_matrix(
+        B, upper_triangular=upper_triangular
+    )
     np.testing.assert_almost_equal(rotate, rotate_B)
     np.testing.assert_almost_equal(scale, scale_B)
     np.testing.assert_almost_equal(shear, shear_B)
@@ -35,3 +44,29 @@ def test_shear_matrix_from_angle():
     matrix = shear_matrix_from_angle(35)
     np.testing.assert_almost_equal(np.diag(matrix), [1] * 3)
     np.testing.assert_almost_equal(matrix[-1, 0], np.tan(np.deg2rad(55)))
+
+
+upper = np.array([[1, 1], [0, 1]])
+lower = np.array([[1, 0], [1, 1]])
+full = np.array([[1, 1], [1, 1]])
+
+
+def test_is_matrix_upper_triangular():
+    """Test if a matrix is upper triangular."""
+    assert is_matrix_upper_triangular(upper)
+    assert not is_matrix_upper_triangular(lower)
+    assert not is_matrix_upper_triangular(full)
+
+
+def test_is_matrix_lower_triangular():
+    """Test if a matrix is lower triangular."""
+    assert not is_matrix_lower_triangular(upper)
+    assert is_matrix_lower_triangular(lower)
+    assert not is_matrix_lower_triangular(full)
+
+
+def test_is_matrix_triangular():
+    """Test if a matrix is triangular."""
+    assert is_matrix_triangular(upper)
+    assert is_matrix_triangular(lower)
+    assert not is_matrix_triangular(full)
diff --git a/napari/utils/transforms/_tests/test_transforms.py b/napari/utils/transforms/_tests/test_transforms.py
--- a/napari/utils/transforms/_tests/test_transforms.py
+++ b/napari/utils/transforms/_tests/test_transforms.py
@@ -240,3 +240,24 @@ def test_affine_matrix_inverse(dimensionality):
     np.testing.assert_almost_equal(
         transform.inverse.affine_matrix, np.linalg.inv(A)
     )
+
+
+def test_repeat_shear_setting():
+    """Test repeatedly setting shear with a lower triangular matrix."""
+    # Note this test is needed to check lower triangular
+    # decomposition of shear is working
+    mat = np.eye(3)
+    mat[2, 0] = 0.5
+    transform = Affine(shear=mat.copy())
+    # Check shear decomposed into lower triangular
+    np.testing.assert_almost_equal(mat, transform.shear)
+
+    # Set shear to same value
+    transform.shear = mat.copy()
+    # Check shear still decomposed into lower triangular
+    np.testing.assert_almost_equal(mat, transform.shear)
+
+    # Set shear to same value
+    transform.shear = mat.copy()
+    # Check shear still decomposed into lower triangular
+    np.testing.assert_almost_equal(mat, transform.shear)