Repository: BBQuercus/deepBlink (language: python)

Issue:
How to fine-tune with my custom data?
Hello there! Thanks for this tool. Its not only great, but having a simple and clear description of everything makes it very convenient to use.

I am trying to achieve a model that works accurately on FISH data. So far I've tested the included _smhfish1.h5_ and also trained my own model file by using the annotation that I made on my custom data. My issue is that the included smhfish1.h5 model works better than the one Ive trained, however its still not superb, just acceptable.

So I was wondering if there is another (simple way) where I could refine the _smhfish1.h5_ weights to get a better result on my own data.

Also, is there a way or any built in method that is capable of measuring an average/overall accuracy (like calculating the mAP value)?

Thank you!



Discussion / hints:
Hey. Thanks for the feedback.

You can use the pre-trained model that you downloaded and just pass the path in to the training config (`train_args` -> `pre_train`).

For scoring we used the F1 integral score described in the paper which is accessible in python through `deepblink.metrics.f1_integral`. More easily though, you can use `deepblink.metrics.compute_metrics` which will list out everything a bit nicer.
Hi @BBQuercus. Thanks for the quick reply, I really appreciate it!

I've tried to use the mentioned pretrained network, but as it seems, the result are not getting better.
With the default (0.5) threshold I was getting totally wrong results, see

![Screenshot from 2022-03-01 17-11-50](https://user-images.githubusercontent.com/3785204/156241402-a3503e24-1f62-41e1-b74b-f180777ab09e.png)

Zooming into it:

![Screenshot from 2022-03-01 17-12-01](https://user-images.githubusercontent.com/3785204/156241446-2af32aff-09f6-419e-b4f9-4b3006bf328d.png)


When I set the threshold to 0.6, I am receiving less predicted dots, however they are not accurate at all:

![Screenshot from 2022-03-01 17-09-15](https://user-images.githubusercontent.com/3785204/156241610-b1feb648-6cb3-40ca-8c2b-42efcf097145.png)
![Screenshot from 2022-03-01 17-08-23](https://user-images.githubusercontent.com/3785204/156241651-3c9bcb96-f46f-427f-be7d-6c894a4547a7.png)

I have nearly 5.000 training examples and these predictions were made on the training samples. However still, the downloaded model (mentioned above) works much better on them, than the refined one. Have you ever experienced something similar? Do you have maybe any ideas why is this happening? 

Thanks!

Hey, thanks for the feedbacks.

Can you share your dataset (or subset) and your config file for training?

Thanks
Hello @zhanyinx!

I've copied 4 images and their corresponding csvs (generated with the recommended TrackMate - I was taking care not to have false positives, but rather false negatives in the labelling set), together with the used config file to my google-drive. https://drive.google.com/drive/folders/1ZzXhtRDd5BsPWaEJ5A4mwJMuETtlohbq?usp=sharing 
I did not want to share my data publicly so please request there an access. Thanks!
We had a look at your data and think to have found the issue. If you use raw microscopy data as input in some cases it'll save metadata on what the microns/pixel size is. TrackMate then uses that information and doesn't allow you to change it. Since deepBlink doesn't really care about the "real" size of stuff, the labels will be wrong if not everything is labeled in pixel-dimensions. The easiest way we found to remove this micron scale is in Fiji via `Analyze` -> `Set scale...` -> `Click to remove scale` before loading it into TrackMate. Alternatively you can also use python to load/save images (e.g. using `skim age.io.imread/imsave`).

Regarding your already labeled images – if you have the same sizes everywhere (i.e. not multiple magnifications), the easiest would be to load everything as data frame in pandas (let me know if you're familiar with pandas) and multiply the `POSITION_X` and `POSITION_Y` columns by the constant that converts microns back to pixels.

We'll make sure to add this rather important information of reseting scale as comment in our video (currently it was only on our wiki).
@BBQuercus how about adding an option to deepblink to account for scaling in the spot coordinates for training? TrackMate always made a great effort to support real world (scaled) coordinates throughout the analysis, because it's important that track dynamics (like mean squared displacement MSD, etc.) are measured in a scientifically meaningful way (in particular on anisotropic image data, of course).

It would be a shame if people have to keep working around this by removing the scale before running TrackMate, and not be able to use their already tracked data in an easy way. And I think it wouldn't be too much of an effort to implement a scaling function into the training process. What do you think?

@imagejan I guess we can add it to the dataset creation stage to keep the actual training process intact, especially between different datasets. I'll have to see how easy it is to extract the scale from the images metadata (any experience on that?) or just require the user to provide a fixed scaling constant / pixel size. At inference time it would be the same story.
Thanks for the explanation @BBQuercus. Actually in my dataset, I have over 60 channels (yesterday I sent you only 2 of those) and the spot-size on all of them will likely differ (however they remain still very small, pixel-wise).  As much as I understand, scaling would be indeed a solution - just imagine having 60 types of spots with different sizes.

Regarding the solution for those 2 channels I sent you yesterday, to do a new labelling on them would not be a big deal, if only removing the scale would help the training. In that case, would it be necessary to do a workaround/modification also on the test images? (I also want to note, that the original images are quite big, over 6000x6000px, but I am playing around with cropped parts just to speed up the process and test whether the approach works well - maybe its a useful info.)
The scaling was referring to the x/y positions and not the spot size. The problem arises since the pixel size isn't also 1 micron (but a lot smaller). Therefore, when visualising spots using pixel dimensions, everything will get clustered in the top left corner of the image instead of being spread out evenly. So scaling would multiply the x/y positions by a constant to make sure everything starts at 0 and ends where the image ends too.

We'll probably add an easier way to visualise a dataset (and prediction) soon to view what the model "sees".

Fix patch:
diff --git a/deepblink/cli/_argparse.py b/deepblink/cli/_argparse.py
index 9e3d19b..4bf57e8 100644
--- a/deepblink/cli/_argparse.py
+++ b/deepblink/cli/_argparse.py
@@ -8,6 +8,7 @@
 from ._parseutil import FileFolderType
 from ._parseutil import FileType
 from ._parseutil import FolderType
+from ._parseutil import PixelSizeType
 from ._parseutil import ProbabilityType
 from ._parseutil import ShapeType
 from ._parseutil import _add_utils
@@ -123,7 +124,7 @@ def _parse_args_create(
         help=(
             "Path to the directory containing labels in csv format. "
             "The default path accounts for using the FIJI macro described on the wiki. "
-            "[default: --INPUT/labels/]"
+            "[default: --INPUT/]"
         ),
     )
     group2.add_argument(
@@ -150,6 +151,19 @@ def _parse_args_create(
             "[default: 512]"
         ),
     )
+    group2.add_argument(
+        "-ps",
+        "--pixel-size",
+        default=None,
+        type=PixelSizeType(),
+        help=(
+            "Pixel size in micrometer. "
+            "If given, will convert all labels to pixel coordinates. "
+            "Can be a single value or a tuple of two values for x and y, respectively. "
+            "If only one value is given, it will be used for both x and y. "
+            "[default: 1]"
+        ),
+    )
     group2.add_argument(
         "-m",
         "--minspots",
diff --git a/deepblink/cli/_create.py b/deepblink/cli/_create.py
index c011ff1..e8f1222 100644
--- a/deepblink/cli/_create.py
+++ b/deepblink/cli/_create.py
@@ -1,6 +1,6 @@
 """CLI submodule for creating a new dataset."""
 
-from typing import List, Tuple
+from typing import List, Tuple, Union
 import logging
 import os
 
@@ -12,6 +12,7 @@
 from ..io import basename
 from ..io import grab_files
 from ..io import load_image
+from ..util import predict_pixel_size
 from ..util import train_valid_split
 
 
@@ -23,6 +24,7 @@ class HandleCreate:
         arg_labels: Path to folder with labels.
         arg_name: Name of dataset file to be saved.
         arg_size: Size of image to be cropped.
+        arg_pixel_size: Pixel size of images.
         arg_testsplit: Test vs. Trainval split percentage.
         arg_testsplit: Valid vs. Train split percentage.
         logger: Logger to log verbose output.
@@ -37,6 +39,7 @@ def __init__(
         arg_labels: str,
         arg_name: str,
         arg_size: int,
+        arg_pixel_size: Union[float, Tuple[float, float]],
         arg_testsplit: int,
         arg_validsplit: int,
         arg_minspots: int,
@@ -46,6 +49,7 @@ def __init__(
         self.raw_labels = arg_labels
         self.raw_name = arg_name
         self.img_size = arg_size
+        self.pixel_size = arg_pixel_size
         self.test_split = arg_testsplit
         self.valid_split = arg_validsplit
         self.minspots = max(1, arg_minspots)
@@ -63,15 +67,15 @@ def __call__(self):
         self.logger.info(f"\U0001F3C1 dataset created at {self.fname_out}")
 
     @property
-    def abs_labels(self):
+    def abs_labels(self) -> str:
         """Return absolute path to directory with labels."""
         # Full path name
         if self.raw_labels is not None:
             path = os.path.abspath(self.raw_labels)
             self.logger.debug(f"using provided label path at {path}")
-        # Default path with labels/ subdirectory
-        elif os.path.isdir(os.path.join(self.abs_input, "labels")):
-            path = os.path.join(self.abs_input, "labels")
+        # Default path in input directory
+        elif os.path.isdir(self.abs_input):
+            path = self.abs_input
             self.logger.debug(f"using default label path at {path}")
         else:
             self.logger.debug(
@@ -80,13 +84,13 @@ def abs_labels(self):
             raise ValueError(
                 (
                     "\U0000274C No label path found.\t"
-                    "Please use a directory called 'labels' in input or use the '--labels' flag."
+                    "Please provide labels in the input directory or use the '--labels' flag."
                 )
             )
         return path
 
     @property
-    def fname_out(self):
+    def fname_out(self) -> str:
         """Return the absolute path to the dataset."""
         if self.raw_name is not None:
             # Full path and file given
@@ -115,9 +119,33 @@ def fname_out(self):
             self.logger.debug(f"using default output at {path}")
         return path
 
+    def get_pixel_size(self, image: str) -> Tuple[float, float]:
+        """Return the pixel size of an image."""
+        # Use user-provided pixel size
+        if self.pixel_size is not None:
+            self.logger.debug(f"using provided pixel size {self.pixel_size}")
+            if isinstance(self.pixel_size, tuple):
+                return self.pixel_size
+            return (self.pixel_size, self.pixel_size)
+
+        # Try to predict pixel size
+        try:
+            size = predict_pixel_size(image)
+            self.logger.debug(f"using predicted pixel size {size}")
+            return size
+        except (ValueError, KeyError, IndexError) as e:
+            self.logger.warning(
+                f"\U000026A0 {e} encountered. Pixel size for image {image} could not be predicted."
+            )
+
+        self.logger.debug(f"defaulting pixel size to {size}")
+        return (1.0, 1.0)
+
     @property
-    def image_label_lists(self):
-        """Return lists with all images and labels."""
+    def image_label_size_lists(
+        self,
+    ) -> Tuple[List[np.ndarray], List[pd.DataFrame], List[Tuple[float, float]]]:
+        """Return lists with all images, labels, and sizes."""
         fname_images = grab_files(self.abs_input, self.extensions)
         fname_labels = grab_files(self.abs_labels, extensions=("csv",))
         self.logger.debug(f"images - found {len(fname_images)} files: {fname_images}")
@@ -125,6 +153,7 @@ def image_label_lists(self):
 
         images = []
         labels = []
+        sizes = []
         for image, label in zip(fname_images, fname_labels):
             if basename(image) != basename(label):
                 self.logger.warning(
@@ -145,23 +174,28 @@ def image_label_lists(self):
 
             # Read image
             img = load_image(image, is_rgb=False)
-            if all(shape >= self.img_size for shape in img.shape):
-                images.append(img)
-                labels.append(df)
-            else:
+            if not all(shape >= self.img_size for shape in img.shape):
                 self.logger.warning(
                     f"\U000026A0 image {image} was too small! {img.shape} < {self.img_size}"
                 )
+                continue
+
+            size = self.get_pixel_size(image)
+            images.append(img)
+            labels.append(df)
+            sizes.append(size)
 
         if not images:
             raise ValueError(
                 "No images matched the format criteria. Please check image size and labelling."
             )
         self.logger.debug(f"using {len(images)} non-empty files")
-        return images, labels
+        return images, labels, sizes
 
     @staticmethod
-    def convert_labels(image: np.ndarray, df: pd.DataFrame) -> pd.DataFrame:
+    def convert_labels(
+        image: np.ndarray, df: pd.DataFrame, pixel_size: Tuple[float, float]
+    ) -> pd.DataFrame:
         """Pre-processes labels to be used in deepBlink.
 
         Renames X/Y to c/r respectively for easier handling with rearrangement to r/c.
@@ -188,6 +222,10 @@ def convert_labels(image: np.ndarray, df: pd.DataFrame) -> pd.DataFrame:
             df[name] = df[name].where(df[name] < var, var)
             df[name] = df[name].where(df[name] > 0, 0)
 
+        # Scale coordinates to pixel size
+        size_x, size_y = pixel_size
+        df["r"] = df["r"] / size_y
+        df["c"] = df["c"] / size_x
         return df
 
     def crop_image(
@@ -239,8 +277,10 @@ def get_file_lists(self):
         self.image_list = []
         self.label_list = []
 
-        for image, label in zip(*self.image_label_lists):
-            label_normalized = self.convert_labels(image=image, df=label)
+        for image, label, size in zip(*self.image_label_size_lists):
+            label_normalized = self.convert_labels(
+                image=image, df=label, pixel_size=size
+            )
             image_crops, label_crops = self.crop_image(image=image, df=label_normalized)
             self.image_list.extend(image_crops)
             self.label_list.extend(label_crops)
@@ -262,9 +302,9 @@ def train_val_test_split(self):
         )
 
         # Convert DataFrame to Numpy array
-        self.y_train = [y.values for y in y_train]
-        self.y_valid = [y.values for y in y_valid]
-        self.y_test = [y.values for y in y_test]
+        self.y_train = np.array([y.values for y in y_train], dtype=object)
+        self.y_valid = np.array([y.values for y in y_valid], dtype=object)
+        self.y_test = np.array([y.values for y in y_test], dtype=object)
 
     def save_npz(self):
         """Save dataset splits as single npz file."""
diff --git a/deepblink/cli/_main.py b/deepblink/cli/_main.py
index 2b2efd8..7ad0c43 100644
--- a/deepblink/cli/_main.py
+++ b/deepblink/cli/_main.py
@@ -74,6 +74,7 @@ def main():
             arg_labels=args.labels,
             arg_name=args.name,
             arg_size=args.size,
+            arg_pixel_size=args.pixel_size,
             arg_testsplit=args.testsplit,
             arg_validsplit=args.validsplit,
             arg_minspots=args.minspots,
diff --git a/deepblink/cli/_parseutil.py b/deepblink/cli/_parseutil.py
index ef6b7be..b9bf00e 100644
--- a/deepblink/cli/_parseutil.py
+++ b/deepblink/cli/_parseutil.py
@@ -56,6 +56,26 @@ def __call__(self, value):  # noqa: D102
         return value
 
 
+class PixelSizeType:
+    """Custom type for pixel size."""
+
+    def __init__(self):
+        pass
+
+    def __call__(self, value):  # noqa: D102
+        if isinstance(value, float):
+            return value
+        if (
+            isinstance(value, tuple)
+            and len(value) == 2
+            and all([isinstance(e, float) for e in value])
+        ):
+            return value
+        raise argparse.ArgumentTypeError(
+            f"Pixel size must be a float or a tuple of two floats. '{value}' is not."
+        )
+
+
 class ShapeType:
     """Custom type for image shapes."""
 
diff --git a/deepblink/util.py b/deepblink/util.py
index a96bfb9..a4a912d 100755
--- a/deepblink/util.py
+++ b/deepblink/util.py
@@ -2,8 +2,11 @@
 
 from typing import Callable, Iterable, Tuple, Union
 import importlib
+import os
 import random
 
+from PIL import Image
+from PIL.TiffTags import TAGS
 import numpy as np
 import pandas as pd
 
@@ -127,3 +130,25 @@ def predict_shape(shape: tuple) -> str:
     sorted_dims = [k for k, v in sorted(dims.items(), key=lambda item: item[1])]
     order = ",".join(sorted_dims)
     return order
+
+
+def predict_pixel_size(fname: str) -> Tuple[float, float]:
+    """Predict the pixel size based on tifffile metadata."""
+    if not fname.endswith(".tif"):
+        raise ValueError(f"{fname} is not a tif file.")
+    if not os.path.isfile(fname):
+        raise ValueError(f"{fname} does not exist.")
+
+    image = Image.open(fname)
+    if len(image.size) != 2:
+        raise ValueError(f"Image {fname} has more than 2 dimensions.")
+
+    # Get resolutions from tiff metadata
+    meta_dict = {TAGS[key]: image.tag[key] for key in image.tag_v2}
+    x_res = meta_dict.get("XResolution", ((1, 1),))
+    y_res = meta_dict.get("YResolution", ((1, 1),))
+    unit = meta_dict.get("ResolutionUnit", (1,))
+    x_size = x_res[0][1] / x_res[0][0] * unit[0]
+    y_size = y_res[0][1] / y_res[0][0] * unit[0]
+
+    return x_size, y_size


Test patch:
diff --git a/tests/data/image_micron.tif b/tests/data/image_micron.tif
new file mode 100644
index 0000000..b0b199d
Binary files /dev/null and b/tests/data/image_micron.tif differ
diff --git a/tests/data/image_pixel.tif b/tests/data/image_pixel.tif
new file mode 100644
index 0000000..4d034d8
Binary files /dev/null and b/tests/data/image_pixel.tif differ
diff --git a/tests/test_cli.py b/tests/test_cli.py
index c82f917..d7c8244 100644
--- a/tests/test_cli.py
+++ b/tests/test_cli.py
@@ -43,6 +43,7 @@ def test_create():
         arg_labels="None",
         arg_name="None",
         arg_size=0,
+        arg_pixel_size=1.0,
         arg_testsplit=0,
         arg_validsplit=0,
         arg_minspots=0,
@@ -59,7 +60,7 @@ def test_create():
 
     for label in labels:
         df = pd.read_csv(label, index_col=0)
-        df = handler.convert_labels(image, df)
+        df = handler.convert_labels(image, df, (1.0, 1.0))
         assert df.columns.to_list() == ["r", "c"]
         assert df.dtypes.to_list() == [np.float64, np.float64]
         assert df["r"].min() >= 0
diff --git a/tests/test_util.py b/tests/test_util.py
index c74e166..100ffb7 100644
--- a/tests/test_util.py
+++ b/tests/test_util.py
@@ -1,6 +1,8 @@
 """Unittests for the deepblink.util module."""
 # pylint: disable=missing-function-docstring
 
+import os
+
 from hypothesis import given
 from hypothesis.strategies import floats
 from hypothesis.strategies import lists
@@ -11,6 +13,7 @@
 
 from deepblink.util import delete_non_unique_columns
 from deepblink.util import get_from_module
+from deepblink.util import predict_pixel_size
 from deepblink.util import predict_shape
 from deepblink.util import relative_shuffle
 from deepblink.util import remove_falses
@@ -95,3 +98,12 @@ def test_predict_shape_1(shape, expected):
 def test_predict_shape_2(shape):
     with pytest.raises(ValueError):
         predict_shape(shape)
+
+
+def test_predict_pixel_size():
+    images = [
+        os.path.join(os.path.dirname(__file__), "./data/image_pixel.tif"),
+        os.path.join(os.path.dirname(__file__), "./data/image_micron.tif"),
+    ]
+    assert predict_pixel_size(images[0]) == (1.0, 1.0)
+    assert predict_pixel_size(images[1]) == pytest.approx((0.3878, 0.4824))