An in-depth EfficientNet tutorial using TensorFlow — How to use EfficientNet on a custom dataset. | by Mostafa Ibrahim | Towards Data Science

Convolutional Neural Networks (ConvNets) are commonly developed at a fixed resource budget, and then scaled up for better accuracy if more resources are available. In this paper, we systematically study model scaling and identify that carefully balancing network depth, width, and resolution can lead to better performance. Based on this observation, we propose a new scaling method that uniformly scales all dimensions of depth/width/resolution using a simple yet highly effective compound coefficient. We demonstrate the effectiveness of this method on scaling up MobileNets and ResNet.
Source: arxiv
EfficientNet has been quite a strong one of the state-of-art image classification networks for a while now. I can see it being used quite heavily in Kaggle competitions for image classification with 0.90+ AUC and I thought I would put our a tutorial here since there aren’t that many online.
I won’t be going over the theoretical part of EfficientNet since there are tons of online resources for that, instead, I will be going over the coding bit. You can use efficientNet-pytorch, however, I usually find TensorFlow quicker and easier to use.
The dataset we are going to be using here is a Chest X-ray dataset from the Kaggle competition VinBigData. We will be using a resized version of 512x512 images since the original images are quite huge (2k+). You can find the resized version here. Anyway, the main aim of the tutorial is to for you to use it on a custom dataset.
Along with the images, we have a dataframe that specifies the class_id for each image:
The first thing you want to do is to run
!pip  install tensorflow-gpu
This will allow you to train your model on the GPU (if you have one). Next thing is to import a few packages:
from tensorflow.keras.applications import * #Efficient Net included herefrom tensorflow.keras import modelsfrom tensorflow.keras import layersfrom keras.preprocessing.image import ImageDataGeneratorimport osimport shutilimport pandas as pdfrom sklearn import model_selectionfrom tqdm import tqdmfrom tensorflow.keras import optimizersimport tensorflow as tf#Use this to check if the GPU is configured correctlyfrom tensorflow.python.client import device_libprint(device_lib.list_local_devices())
To go even further, we use neural architecture search to design a new baseline network and scale it up to obtain a family of models, called EfficientNets, which achieve much better accuracy and efficiency than previous ConvNets. In particular, our EfficientNet-B7 achieves state-of-the-art 84.3% top-1 accuracy on ImageNet, while being 8.4x smaller and 6.1x faster on inference than the best existing ConvNet. Our EfficientNets also transfer well and achieve state-of-the-art accuracy on CIFAR-100 (91.7%), Flowers (98.8%), and 3 other transfer learning datasets, with an order of magnitude fewer parameters. Source code is at https: //github.com/tensorflow/tpu/tree/ master/models/official/efficientnet
Source: arxiv
Okay next thing we need to do is to set up efficientNet and install pre-trained weights
# Options: EfficientNetB0, EfficientNetB1, EfficientNetB2, EfficientNetB3, ... up to  7# Higher the number, the more complex the model is. and the larger resolutions it  can handle, but  the more GPU memory it will need# loading pretrained conv base model#input_shape is (height, width, number of channels) for imagesconv_base = EfficientNetB6(weights="imagenet", include_top=False, input_shape=input_shape)
Weights=”imagenet” allows us to do transfer learning, but you can set it to None if you want (you probably shouldn’t do this). include_top=False allows us to easily change the final layer to our custom dataset.
After installing the model, we want to do a small bit of configuration to make it suitable for our custom dataset:
model = models.Sequential()model.add(conv_base)model.add(layers.GlobalMaxPooling2D(name="gap"))#avoid overfittingmodel.add(layers.Dropout(dropout_rate=0.2, name="dropout_out"))# Set NUMBER_OF_CLASSES to the number of your final predictions.model.add(layers.Dense(NUMBER_OF_CLASSES, activation="softmax", name="fc_out"))conv_base.trainable = False
The model is prepared. Now we need to prepare the dataset. We are going to be using a flow_from_directory along with Keras’s ImageDataGenerator. This method will be expecting training and validation directories. In each directory, there should be a separate directory for each class with the corresponding images under that directory.
To start with, let’s create a directory for each class under training & validation.
TRAIN_IMAGES_PATH = './vinbigdata/images/train' #12000VAL_IMAGES_PATH = './vinbigdata/images/val' #3000External_DIR = '../input/vinbigdata-512-image-dataset/vinbigdata/train' # 15000os.makedirs(TRAIN_IMAGES_PATH, exist_ok = True)os.makedirs(VAL_IMAGES_PATH, exist_ok = True)classes = [ 'Aortic enlargement',            'No Finding']# Create directories for each class.for class_id in [x for x in range(len(classes))]:    os.makedirs(os.path.join(TRAIN_IMAGES_PATH, str(class_id)), exist_ok = True)    os.makedirs(os.path.join(VAL_IMAGES_PATH, str(class_id)), exist_ok = True)
The next thing to do is to copy each image to its correct directory:
Input_dir = '/kaggle/input/vinbigdata-512-image-dataset/vinbigdata/train'def preproccess_data(df, images_path):    for column, row in tqdm(df.iterrows(), total=len(df)):        class_id = row['class_id']        shutil.copy(os.path.join(Input_dir, f"{row['image_id']}.png"), os.path.join(images_path, str(class_id)))df = pd.read_csv('../input/vinbigdata-512-image-dataset/vinbigdata/train.csv')df.head()#Split the dataset into 80% training and 20% validationdf_train, df_valid = model_selection.train_test_split(df, test_size=0.2, random_state=42, shuffle=True)#run the  function on each of thempreproccess_data(df_train, TRAIN_IMAGES_PATH)preproccess_data(df_valid, VAL_IMAGES_PATH)
Now, you can check the dataset directory and all of the images should be copied to their correct sub-directories. The next step is to pass the dataset to the generator and then start training:
# I love the  ImageDataGenerator class, it allows us to specifiy whatever augmentations we want so easily...train_datagen = ImageDataGenerator(    rescale=1.0 / 255,    rotation_range=40,    width_shift_range=0.2,    height_shift_range=0.2,    shear_range=0.2,    zoom_range=0.2,    horizontal_flip=True,    fill_mode="nearest",)# Note that the validation data should not be augmented!#and a very important step is to normalise the images through  rescalingtest_datagen = ImageDataGenerator(rescale=1.0 / 255)train_generator = train_datagen.flow_from_directory(    # This is the target directory    TRAIN_IMAGES_PATH,    # All images will be resized to target height and width.    target_size=(height, width),    batch_size=batch_size,    # Since we use categorical_crossentropy loss, we need categorical labels    class_mode="categorical",)validation_generator = test_datagen.flow_from_directory(    VAL_IMAGES_PATH,    target_size=(height, width),    batch_size=batch_size,    class_mode="categorical",)model.compile(    loss="categorical_crossentropy",    optimizer=optimizers.RMSprop(lr=2e-5),    metrics=["acc"],)
If all goes according to plan, you should get a similar message to this:
Found X images belonging to x classes.Found Y images belonging to x classes.
history = model.fit_generator(    train_generator,    steps_per_epoch=NUMBER_OF_TRAINING_IMAGES // batch_size,    epochs=epochs,    validation_data=validation_generator,    validation_steps=NUMBER_OF_VALIDATION_IMAGES // batch_size,    verbose=1,    use_multiprocessing=True,    workers=4,)
y_pred = model.predict(X_test)score = model.evaluate(X_test, y_test,verbose=1)
# Import the modules from `sklearn.metrics` from sklearn.metrics import confusion_matrix, precision_score, recall_score, f1_score# Confusion matrix confusion_matrix(y_test, y_pred) precision_score(y_test, y_pred) recall_score(y_test, y_pred) f1_score(y_test,y_pred) 
The next part is to further evaluate the model, there are a lot of resources for doing this and since you will probably be interested in exploring tons of different metrics, this should be quite easy using Keras.
A few possible improvements on the tutorial here is to use cross-validation by creating several folds and then ensembling the final predictions. Furthermore, you can use more advanced data augmentation techniques such as Mixup, Cutup, and Jitter.