Visualizing and Preprocessing Image Dataset | by Himanshu Sharma | Towards Data Science

When creating computer vision projects or models, we usually work with large datasets. Not only are image datasets larger in size than text datasets, but there are more images used so that our model does not overfit and has a high performance.
Where can we find new, fresh datasets? Is downloading possible, given their size? We all have worked on datasets like MNIST, FashionMNIST, CIFAR-10, etc., but these are old and have been used many times. Once we access new datasets, how can we use them easily without downloading them locally?
The answer is to use TensorBay from Graviti. TensorBay is a dataset management platform that automates and streamlines the entire process, including data hosting, version control, and data visualization. For easy integration, Graviti has developed a Python-based SDK for TensorBay that performs these services.
For visualization of local datasets, we can use Pharos, which is a plugin of TensorBay SDK. Pharos launches a Flask application that allows us to visualize the data and also check its annotations. These visualizations help in identifying how our data is organized and whether or not there is an anomaly in the data.
For image processing, Skimage, a Python library, is used. Skimage contains different algorithms that can be used to edit images.
In this article, we will explore image data visualization using TensorBay, Pharos, and Skimage. Let’s get started...
Before visualizing the data you need to have an access key to access the data and fork the dataset. Along with visualization, this key will give you access to other features like version control, dataset creation, etc.
account.graviti.com
The link given above can be used to create an account on Graviti. After signing up you can generate and use your access key from the Developer Tools section.
The next step is to fork the datasets. Graviti provides 1000+ datasets that can be easily forked and used accordingly. You can select any dataset here. For this article, we are using the 17 Category Flower dataset.
To visualize a dataset, land on its homepage in TensorBay and use Pharos to launch a Flask application.
Create a Jupyter Notebook
The final step is to create a Jupyter Notebook that will be used to load the forked dataset. But first, we need to install the required TensorBay and its plugin Pharos Python libraries using the commands given below.
In the code shown below, the dataset is being read on our local machine and images fetched from the URL to save them in a list that will be used for preprocessing. Remember, you need to provide your own access key and fork the data you want to use.
from tensorbay import GASfrom tensorbay.dataset import Segmentfrom tensorbay.dataset import Data, Datasetimport requestsfrom PIL import ImageACCESS_KEY = “Your Access Key”gas = GAS(ACCESS_KEY)dataset_client = gas.get_dataset(“Flower17–1”)img_list=[]segments = dataset_client.list_segment_names()segment = Segment(“default”, dataset_client)for data in segment:   fp = data.open()   img_list.append(Image.open(requests.get(fp.url,            stream=True).raw))img_list
In the image above you can see how we can load different images from the URL. Now we will visualize the images we will work on, and then preprocess those images, which are stored on a list.
plt.figure(figsize=(11,11))for i in range(1,10):plt.subplot(3,3,i)plt.tight_layout()plt.imshow(img_list[i-1])plt.title(‘Flower’)plt.xlabel(img_list[i-1].shape[1], fontsize=10)plt.ylabel(img_list[i-1].shape[0], fontsize=10)
#converting image to numpy arrayimport numpy as npimage1 = np.array(img_list[0])image1
image1.shape
image1.size
#Converting Grayscalefrom skimage.color import rgb2graygrayscale = rgb2gray(img_list[0])#Edge Detectionfrom skimage import filtersed_sobel = filters.sobel(grayscale)imshow(ed_sobel, cmap=’gray’);
from skimage.exposure import histogramhist, hist_centers = histogram(grayscale)#Plotting the Image and the Histogram of gray valuesfig, axes = plt.subplots(1, 2, figsize=(8, 3))axes[0].imshow(grayscale, cmap=plt.cm.gray)axes[0].axis(‘off’)axes[1].plot(hist_centers, hist, lw=2)axes[1].set_title(‘histogram of gray values’)
Try this process with different datasets that are available at Graviti, and create visualizations using Pharos. Please let me know how it turned out by leaving your comments in the response section.