Repository: DigitalSlideArchive/HistomicsStream (language: python)

Issue:
Demonstrate application of a useful histopathology model.
`tensorflow_stream.ipynb` should demonstrate application of a useful histopathology model.  (This is the part of Issue #42 that was not completed with Pull Request #43.)


Discussion / hints:

Fix patch:
diff --git a/example/tensorflow_stream.ipynb b/example/tensorflow_stream.ipynb
index 870aebf..576538a 100644
--- a/example/tensorflow_stream.ipynb
+++ b/example/tensorflow_stream.ipynb
@@ -1,18 +1,103 @@
 {
  "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "dc04ec63-29d0-4d8c-b9a2-00a15bcdfb98",
+   "metadata": {},
+   "source": [
+    "# Demonstration of histomics_stream\n",
+    "\n",
+    "Click to open in [[GitHub](https://github.com/DigitalSlideArchive/histomics_stream/tree/master/example/tensorflow_stream.ipynb)] [[Google Colab](https://colab.research.google.com/github/DigitalSlideArchive/histomics_stream/blob/master/example/tensorflow_stream.ipynb)]\n",
+    "\n",
+    "The `histomics_stream` Python package sits at the start of any machine learning workflow that is built on the TensorFlow machine learning library.  The package is responsible for efficient access to the input image data that will be used to fit a new machine learning model or will be used to predict regions of interest in novel inputs using an already learned model."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "e292dd77-6a28-49b0-b881-b8069e939381",
+   "metadata": {},
+   "source": [
+    "## Installation\n",
+    "\n",
+    "If you are running this notebook on Google Colab or another system where `histomics_stream` is not yet installed then it can be installed with the following command."
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "d1ecc019-0935-4d07-a245-f61ca1836675",
+   "id": "58dfc688-55f0-41f8-b497-f1715e1c48cc",
    "metadata": {},
    "outputs": [],
    "source": [
-    "import copy\n",
-    "import histomics_stream as hs\n",
+    "!pip install histomics_stream"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "b965f4e7-7384-44fd-b6c8-db34a9ee5bea",
+   "metadata": {},
+   "source": [
+    "## Fetching and creating the test data\n",
+    "This notebook has demonstrations that use the files `TCGA-3L-AA1B-01Z-00-DX1.8923A151-A690-40B7-9E5A-FCBEDFC2394F.svs` (1.2 GB) and `TCGA-3L-AA1B-01Z-00-DX1.8923A151-A690-40B7-9E5A-FCBEDFC2394F-mask.png` (28 kB),  If we don't already have them then the first is fetched and the second is randomly created."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "51185999-5b64-465b-9eac-39a66878aafa",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "\n",
+    "remote_filename = \"https://tiatoolbox.dcs.warwick.ac.uk/sample_wsis/TCGA-3L-AA1B-01Z-00-DX1.8923A151-A690-40B7-9E5A-FCBEDFC2394F.svs\"\n",
+    "local_filename = remote_filename.split(\"/\")[-1]\n",
+    "if not os.path.exists(local_filename):\n",
+    "    print(f\"Downloading {remote_filename} ...\")\n",
+    "    import requests\n",
+    "    import shutil\n",
     "\n",
-    "# import tensorflow as tf\n",
+    "    # This does not decompress zip files or anything like that.\n",
+    "    with requests.get(remote_filename, stream=True) as r:\n",
+    "        with open(local_filename, \"wb\") as f:\n",
+    "            shutil.copyfileobj(r.raw, f)\n",
+    "print(f\"Have {local_filename}.\")\n",
     "\n",
-    "print(\"Configuring with histomics_stream\")\n",
+    "# Create a random mask for the image if necessary\n",
+    "mask_filename = os.path.splitext(local_filename)[0] + \"-mask.png\"\n",
+    "if not os.path.exists(mask_filename):\n",
+    "    import large_image\n",
+    "    print(f\"Creating {mask_filename} ...\")\n",
+    "    ts = large_image.open(local_filename)\n",
+    "    import numpy as np\n",
+    "    from PIL import Image\n",
+    "\n",
+    "    arr = np.random.randint(0, 2, (ts.sizeY // 256, ts.sizeX // 256), dtype=np.int8)\n",
+    "    im = Image.fromarray(arr)\n",
+    "    im.save(mask_filename)\n",
+    "print(f\"Have {mask_filename}.\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "1c9ec1a1-899f-4d6f-bc26-d15c996c7eb7",
+   "metadata": {},
+   "source": [
+    "## Creating a study for use with histomics_stream\n",
+    "\n",
+    "We describe the input and desired parameters using standard Python lists and dictionaries.  Here we give a high-level configuration; selection of tiles is done subsequently. "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "d5f8d7f8-79bc-471d-83d4-80dcf8eaa78e",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import copy\n",
+    "import histomics_stream as hs\n",
+    "import tensorflow\n",
     "\n",
     "# Create a study and insert study-wide information\n",
     "my_study0 = {\"version\": \"version-1\"}\n",
@@ -22,9 +107,9 @@
     "\n",
     "# Add a slide to the study, including slide-wide information with it.\n",
     "my_slide0 = my_slides[\"Slide_0\"] = {}\n",
-    "my_slide0[\"filename\"] = \"TA232.svs\"\n",
-    "my_slide0[\"slide_name\"] = \"TA232\"\n",
-    "my_slide0[\"slide_group\"] = \"MYC\"\n",
+    "my_slide0[\"filename\"] = local_filename\n",
+    "my_slide0[\"slide_name\"] = \"local_filename_0\"\n",
+    "my_slide0[\"slide_group\"] = \"control\"\n",
     "my_slide0[\"number_pixel_rows_for_chunk\"] = 2048\n",
     "my_slide0[\"number_pixel_columns_for_chunk\"] = 2048\n",
     "\n",
@@ -36,36 +121,48 @@
     ")\n",
     "for slide in my_study0[\"slides\"].values():\n",
     "    find_resolution_for_slide(slide)\n",
-    "print(\"================================================================\")\n",
     "print(f\"my_study0 = {my_study0}\")"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "d4e51b3a-5c43-4d29-a5ea-e35b1f400e37",
+   "metadata": {},
+   "source": [
+    "## Tile selection\n",
+    "\n",
+    "We are going to demonstrate several approaches to choosing tiles.  Each approach will start with its own copy of the `my_study0` that we have built so far."
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "af6100da-943c-4e56-bf6d-d711169daedb",
+   "id": "465dae31-c9ea-42b1-b5f4-31922420d1d0",
    "metadata": {},
    "outputs": [],
    "source": [
-    "# We are going to demonstrate several approaches to choosing tiles.  Each approach will\n",
-    "# start with its own copy of the my_study0 that we have built so far.\n",
-    "\n",
     "# Demonstrate TilesByGridAndMask without a mask\n",
     "my_study_tiles_by_grid = copy.deepcopy(my_study0)\n",
     "tiles_by_grid = hs.configure.TilesByGridAndMask(\n",
     "    my_study_tiles_by_grid,\n",
     "    number_pixel_overlap_rows_for_tile=32,\n",
     "    number_pixel_overlap_columns_for_tile=32,\n",
-    "    randomly_select=100,\n",
+    "    randomly_select=5,\n",
     ")\n",
     "# We could apply this to a subset of the slides, but we will apply it to all slides in\n",
     "# this example.\n",
     "for slide in my_study_tiles_by_grid[\"slides\"].values():\n",
     "    tiles_by_grid(slide)\n",
-    "print(\"================================================================\")\n",
-    "print(\"Finished with TilesByGrid\")\n",
-    "print(f\"my_study_tiles_by_grid = {my_study_tiles_by_grid}\")\n",
-    "\n",
+    "print(f\"my_study_tiles_by_grid = {my_study_tiles_by_grid}\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "08db2853-8503-4c29-bf7f-3301aa258257",
+   "metadata": {},
+   "outputs": [],
+   "source": [
     "# Demonstrate TilesByGridAndMask with a mask\n",
     "my_study_tiles_by_grid_and_mask = copy.deepcopy(my_study0)\n",
     "tiles_by_grid_and_mask = hs.configure.TilesByGridAndMask(\n",
@@ -73,16 +170,22 @@
     "    number_pixel_overlap_rows_for_tile=0,\n",
     "    number_pixel_overlap_columns_for_tile=0,\n",
     "    mask_filename=\"TA232-mask.png\",\n",
-    "    randomly_select=50,\n",
+    "    randomly_select=10,\n",
     ")\n",
     "# We could apply this to a subset of the slides, but we will apply it to all slides in\n",
     "# this example.\n",
     "for slide in my_study_tiles_by_grid_and_mask[\"slides\"].values():\n",
     "    tiles_by_grid_and_mask(slide)\n",
-    "print(\"================================================================\")\n",
-    "print(\"Finished with TilesByGridAndMask\")\n",
-    "print(f\"my_study_tiles_by_grid_and_mask = {my_study_tiles_by_grid_and_mask}\")\n",
-    "\n",
+    "print(f\"my_study_tiles_by_grid_and_mask = {my_study_tiles_by_grid_and_mask}\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "302f0dd8-df08-4d47-a9dc-d1257e0f6579",
+   "metadata": {},
+   "outputs": [],
+   "source": [
     "# Demonstrate TilesByList\n",
     "my_study_tiles_by_list = copy.deepcopy(my_study0)\n",
     "tiles_by_list = hs.configure.TilesByList(\n",
@@ -94,42 +197,146 @@
     "# this example.\n",
     "for slide in my_study_tiles_by_list[\"slides\"].values():\n",
     "    tiles_by_list(slide)\n",
-    "print(\"================================================================\")\n",
-    "print(\"Finished with TilesByList\")\n",
-    "print(f\"my_study_tiles_by_list = {my_study_tiles_by_list}\")\n",
-    "\n",
+    "print(f\"my_study_tiles_by_list = {my_study_tiles_by_list}\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "86e1b285-89ec-442c-bf3b-8caf3c703e4a",
+   "metadata": {},
+   "outputs": [],
+   "source": [
     "# Demonstrate TilesRandomly\n",
     "my_study_tiles_randomly = copy.deepcopy(my_study0)\n",
-    "tiles_randomly = hs.configure.TilesRandomly(my_study_tiles_randomly, randomly_select=3)\n",
+    "tiles_randomly = hs.configure.TilesRandomly(\n",
+    "    my_study_tiles_randomly,\n",
+    "    randomly_select=10,\n",
+    ")\n",
     "# We could apply this to a subset of the slides, but we will apply it to all slides in\n",
     "# this example.\n",
     "for slide in my_study_tiles_randomly[\"slides\"].values():\n",
     "    tiles_randomly(slide)\n",
-    "print(\"================================================================\")\n",
-    "print(\"Finished with TilesRandomly\")\n",
     "print(f\"my_study_tiles_randomly = {my_study_tiles_randomly}\")"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "4134eb9c-9543-46b6-a3fa-40f87e1269be",
+   "metadata": {},
+   "source": [
+    "## Creating a TensorFlow Dataset\n",
+    "\n",
+    "We request tiles indicated by the mask and create a tensorflow Dataset that has the image data for these tiles as well as associated parameters for each tile, such as its location."
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "238948be-b2aa-4558-9c1a-d7dbd7b42cea",
+   "id": "34cac0f7-dd8d-4ec8-aef5-1603fef350ad",
    "metadata": {},
    "outputs": [],
    "source": [
-    "# We choose one of the above examples for further processing.\n",
-    "my_study_of_tiles = my_study_tiles_by_grid\n",
-    "\n",
-    "print(\"Creating a tensorflow Dataset with histomics_stream\")\n",
+    "# Demonstrate TilesByGridAndMask with a mask\n",
+    "my_study_of_tiles = copy.deepcopy(my_study0)\n",
+    "tiles_by_grid_and_mask = hs.configure.TilesByGridAndMask(\n",
+    "    my_study_of_tiles,\n",
+    "    number_pixel_overlap_rows_for_tile=0,\n",
+    "    number_pixel_overlap_columns_for_tile=0,\n",
+    "    mask_filename=\"TA232-mask.png\",\n",
+    "    randomly_select=1000,\n",
+    ")\n",
+    "for slide in my_study_of_tiles[\"slides\"].values():\n",
+    "    tiles_by_grid_and_mask(slide)\n",
+    "print(\"Finished selecting tiles.\")\n",
     "\n",
     "create_tensorflow_dataset = hs.tensorflow.CreateTensorFlowDataset()\n",
     "tiles = create_tensorflow_dataset(my_study_of_tiles)\n",
-    "print(\"Finished with CreateTensorFlowDataset\")\n",
+    "print(\"Finished with CreateTensorFlowDataset\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "14a2379d-9a3f-4463-bd45-036f9922e91b",
+   "metadata": {},
+   "source": [
+    "## Create a model for prediction\n",
+    "\n",
+    "We create a nonsense model to demonstrate how the tensorflow Dataset we just created is used with models.  Note that because each element of our Dataset is a tuple `(rgb_image_data, dictionary_of_annotation)`, a typical model that accepts only the former as its input needs to be wrapped."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "29763ada-b593-4866-a2fc-8cc749486cae",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "num_classes = 3\n",
+    "unwrapped_model = tensorflow.keras.models.Sequential(\n",
+    "    [\n",
+    "        tensorflow.keras.layers.Rescaling(\n",
+    "            1.0 / 255,\n",
+    "            input_shape=(\n",
+    "                my_study_of_tiles[\"number_pixel_rows_for_tile\"],\n",
+    "                my_study_of_tiles[\"number_pixel_columns_for_tile\"],\n",
+    "                3,\n",
+    "            ),\n",
+    "        ),\n",
+    "        tensorflow.keras.layers.Conv2D(16, 3, padding=\"same\", activation=\"relu\"),\n",
+    "        tensorflow.keras.layers.MaxPooling2D(),\n",
+    "        tensorflow.keras.layers.Conv2D(32, 3, padding=\"same\", activation=\"relu\"),\n",
+    "        tensorflow.keras.layers.MaxPooling2D(),\n",
+    "        tensorflow.keras.layers.Conv2D(64, 3, padding=\"same\", activation=\"relu\"),\n",
+    "        tensorflow.keras.layers.MaxPooling2D(),\n",
+    "        tensorflow.keras.layers.Flatten(),\n",
+    "        tensorflow.keras.layers.Dense(128, activation=\"relu\"),\n",
+    "        tensorflow.keras.layers.Dense(num_classes),\n",
+    "    ]\n",
+    ")\n",
+    "\n",
+    "# For this demonstration, we will use the model even though we have not trained it!\n",
+    "\n",
+    "# Each element of the `tiles` tensorflow Dataset is a (rgb_image_data, dictionary_of_annotation) pair.\n",
+    "# Wrap the unwrapped_model so that it knows to use the image.\n",
+    "class WrappedModel(tensorflow.keras.Model):\n",
+    "    def __init__(self, model, *args, **kwargs):\n",
+    "        super(WrappedModel, self).__init__(*args, **kwargs)\n",
+    "        self.model = model\n",
+    "\n",
+    "    def call(self, element):\n",
+    "        return (self.model(element[0]), element[1])\n",
+    "\n",
+    "\n",
+    "model = WrappedModel(unwrapped_model)\n",
+    "print(\"Model built and wrapped.\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "d6abdb76-8fbf-425b-a033-8a0d4a505f96",
+   "metadata": {},
+   "source": [
+    "## Make predictions"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "6c5a910b-ac75-4b74-966a-161beffad661",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import time\n",
     "\n",
-    "print(\"================================================================\")\n",
-    "print(tiles)\n",
-    "# print(\"================================================================\")\n",
-    "# tf.print(tiles)"
+    "print(\"Starting predictions\")\n",
+    "start_time = time.time()\n",
+    "predictions = model.predict(tiles.batch(256))\n",
+    "end_time = time.time()\n",
+    "print(f\"Completed {predictions[0].shape[0]} predictions in {end_time - start_time} s.\")\n",
+    "print(\n",
+    "    f\"Average of {(end_time - start_time) / (predictions[0].shape[0])} s per prediction.\"\n",
+    ")"
    ]
   }
  ],
@@ -149,7 +356,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.8.10"
+   "version": "3.7.5"
   }
  },
  "nbformat": 4,


Test patch:
diff --git a/.github/workflows/build-test-package.yml b/.github/workflows/build-test-package.yml
index a8f58bf..d473b2a 100644
--- a/.github/workflows/build-test-package.yml
+++ b/.github/workflows/build-test-package.yml
@@ -34,7 +34,8 @@ jobs:
 
       - name: Install Python dependencies
         run: |
-          sudo apt install python3-openslide
+          sudo apt update
+          sudo apt install openslide-tools python3-openslide
           python -m pip install --upgrade pip setuptools wheel
           pip install 'flake8${{ matrix.flake8-python-git-tag }}' 'pytest${{ matrix.pytest-python-git-tag }}' 'itk${{ matrix.itk-python-git-tag }}' 'tensorflow${{ matrix.tensorflow-python-git-tag }}'
           pip install 'large-image[all]' --find-links https://girder.github.io/large_image_wheels