Repository: deepchecks/deepchecks

Issue:
[FEAT][USE] Improve simple classification tutorial
**Is your feature request related to a problem? Please describe.**
Simple classification tutorial is simple, but lacks an example of actual value. 

**Describe the solution you'd like**
Change to running the tutorial on a dataset that produces an interesting report, that catches some real outliers / drift / leakage. 

* Make sure to add the dataset used for the tutorial to the package - add the functions for downloading the data and modifying it if needed for the example. 


Discussion:

Resolving patch:
diff --git a/docs/source/tutorials/vision/plot_simple_classification_tutorial.py b/docs/source/tutorials/vision/plot_simple_classification_tutorial.py
--- a/docs/source/tutorials/vision/plot_simple_classification_tutorial.py
+++ b/docs/source/tutorials/vision/plot_simple_classification_tutorial.py
@@ -16,24 +16,32 @@
 """
 
 #%%
-# Defining the data and model
+# Downloading the Data
 # ===========================
 #
-# The data is available from the torch library. We will download and extract it to the current directory.
+# For this example we'll use a small sample of the RGB `EuroSAT dataset <https://github.com/phelber/eurosat#>`_.
+# EuroSAT dataset is based on Sentinel-2 satellite images covering 13 spectral bands and consisting of 10 classes
+# with 27000 labeled and geo-referenced samples.
+
+# Citations:
+#
+# [1] Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. Patrick
+# Helber, Benjamin Bischke, Andreas Dengel, Damian Borth. IEEE Journal of Selected Topics in Applied Earth
+# Observations and Remote Sensing, 2019.
+#
+# [2] Introducing EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification.
+# Patrick Helber, Benjamin Bischke, Andreas Dengel. 2018 IEEE International Geoscience and Remote Sensing Symposium,
+# 2018.
 
 import urllib.request
 import zipfile
-import os
-
-url = 'https://download.pytorch.org/tutorial/hymenoptera_data.zip'
-urllib.request.urlretrieve(url, 'hymenoptera_data.zip')
+import numpy as np
 
-with zipfile.ZipFile('hymenoptera_data.zip', 'r') as zip_ref:
-    zip_ref.extractall('.')
+url = 'https://figshare.com/ndownloader/files/34912884'
+urllib.request.urlretrieve(url, 'EuroSAT_data.zip')
 
-# Rename val folder to test, because the simple classification task expects a test folder.
-if not os.path.exists('hymenoptera_data/test'):
-    os.rename('hymenoptera_data/val', 'hymenoptera_data/test')
+with zipfile.ZipFile('EuroSAT_data.zip', 'r') as zip_ref:
+    zip_ref.extractall('EuroSAT')
 
 #%%
 # Loading a Simple Classification Dataset
@@ -51,9 +59,10 @@
 
 from deepchecks.vision.simple_classification_data import load_dataset
 
-train_ds = load_dataset('hymenoptera_data', train=True, object_type='VisionData', image_extension='jpg')
-test_ds = load_dataset('hymenoptera_data', train=False, object_type='VisionData', image_extension='jpg')
+train_ds = load_dataset('./EuroSAT/euroSAT/', train=True, object_type='VisionData', image_extension='jpg')
+test_ds = load_dataset('./EuroSAT/euroSAT/', train=False, object_type='VisionData', image_extension='jpg')
 
+#%%
 # Running Deepchecks' full suite
 # ==============================
 # That's it, we have just defined the classification data object and are ready to run the train_test_validation suite:
@@ -64,7 +73,7 @@
 result = suite.run(train_ds, test_ds)
 
 #%%
-# Observing the results:
+# Observing the Results:
 # ===========================
 # The results can be saved as a html file with the following code:
 
@@ -74,3 +83,38 @@
 # Or, if working inside a notebook, the output can be displayed directly by simply printing the result object:
 
 result
+
+#%%
+# Understanding the Results:
+# ===========================
+# Looking at the results we see two checks whose conditions have failed:
+#
+# 1. Similar Image Leakage
+# 2. Simple Feature Contribution
+#
+# The first has clearly failed due to the naturally occurring similarity between different ocean / lake image,
+# and the prevailing green of some forest images. We may wish to remove some of these duplicate images but for this
+# dataset they make sense.
+#
+# The second failure is more interesting. The :doc:`Simple Feature Contribution
+# </examples/vision/checks/source/methodology/plot_simple_feature_contribution>` check computes various :doc:`simple
+# image # properties </user-guide/vision/vision_properties>` and checks if the image label can be inferred using a
+# simple model (for example, a Classification Tree) using the property values. The ability to predict the label using
+# these properties is measures by the Predictive Power Score (PPS) and this measure is compared between the training
+# and test dataset. In this case, the condition alerts us to the fact that this PPS for the "RMS Contrast" property was
+# significantly higher in the training dataset than in the test dataset.
+#
+# We'll show the relevant plot again for ease of discussion:
+
+check_idx = np.where([result.results[i].check.name() == 'Simple Feature Contribution'
+                      for i in range(len(result.results))])[0][0]
+result.results[check_idx].display[0]
+
+#%%
+# Here we can see the plot dedicated to the PPS of the property RMS Contrast, which measures the contrast in the
+# image by calculating the grayscale standard deviation of the image. This plot shows us that specifically for the
+# classes "Forest" and "SeaLake" (the same culprits from the Similar Image Leakage condition), the contrast is a
+# great predictor, but only in the training data! This means we have a critical problem - or model may learn to
+# classify these classes using only the contrast, without actually learning anything about the image content. We now
+# can go on and fix this issue (perhaps by adding train augmentations, or enriching our training set) even before we
+# start thinking about what model to train for the task.


Test patch: