The Multi-Channel Neural Network. Neural Networks can, and should, be... | by Eugenio Zuccarelli | Towards Data Science

Neural Networks are widely used across multiple domains, such as Computer Vision, Audio Classification, Natural Language Processing, etc. In most cases, they are considered in each of these domains individually. However, in real-life settings, it is rarely the case that this is the optimal configuration. It is much more common to have multiple channels, meaning several different types of inputs. Similarly to how humans extract insights using a wide range of sensory inputs (audio, visual, etc.), Neural Networks can (and should) be trained on multiple inputs.
Let’s take, for example, the task of emotion recognition.
Humans do not use a single input to effectively classify the interlocutor’s emotion. They do not use, for instance, the facial expressions (visual), the voice (audio), or the meaning of words (text) of the interlocutor only, but a mixture of them. Similarly, Neural Networks can be trained on multiple inputs, such as images, audio and text, processed accordingly (through CNN, NLP, etc.), to come up with an effective prediction of the target emotion. By doing so, Neural Networks can be better able to capture the subtleties that are difficult to get from each channel individually (e.g. sarcasm), similarly to humans.
Considering the task of emotion recognition, that for simplicity we restrict to three classes (Positive, Negative and Neutral), we can think of the system as the following:
The system picks up the audio through a microphone, computes the MEL Spectrogram of the sound as image and transcribes it into a string of text. These two signals are then used as the inputs to the model, each fed to a branch of it. The Neural Network is, indeed, formed by two sections:
The left branch, performing Image Classification through a Convolutional Neural Network
The right branch, performing NLP on the text, using Embeddings.
Finally, the output of each side is fed into a common set of Dense layers, where the last one has three neurons to respectively classify the three classes (Positive, Neutral and Negative).
In this example, we will use the MELD dataset, consisting of short conversations with associated emotion labels, indicating whether the sentiment of the statement is Positive, Neutral or Negative. The dataset can be found here.
You can also find the complete code of this article here.
Right after the sound is collected, the system computes the Spectrogram of the audio signal. The state-of-the-art features for audio classification are MEL Spectrograms and MEL Frequency Cepstral Coefficients. For this example, we will use the MEL Spectrogram.
First of all, we will obviously load the data from a folder containing the audios, split across Training, Validation and Test:
import gensim.models as gmimport glob as gbimport keras.applications as kaimport keras.layers as klimport keras.models as kmimport keras.optimizers as koimport keras_preprocessing.image as kiimport keras_preprocessing.sequence as ksimport keras_preprocessing.text as ktimport numpy as npimport pandas as pdimport pickle as pkimport tensorflow as tfimport utils as ut# DataData_dir = np.array(gb.glob('../Data/MELD.Raw/train_splits/*'))Validation_dir = np.array(gb.glob('../Data/MELD.Raw/dev_splits_complete/*'))Test_dir = np.array(gb.glob('../Data/MELD.Raw/output_repeated_splits_test/*'))# ParametersBATCH = 16EMBEDDING_LENGTH = 32
We can then loop through each audio file in these three folders, compute the MEL Spectrogram and save it as an image into a new folder:
# Convert Audio to Spectrogramsfor file in Data_dir:    filename, name = file, file.split('/')[-1].split('.')[0]    ut.create_spectrogram(filename, name)for file in Validation_dir:    filename, name = file, file.split('/')[-1].split('.')[0]    ut.create_spectrogram_validation(filename, name)for file in Test_dir:    filename, name = file, file.split('/')[-1].split('.')[0]    ut.create_spectrogram_test(filename, name)
To do so, we have created a function in the utilities script for Training, Validation and Test. This function uses the librosa package to load the audio file, process it and then save it as an image:
import librosaimport librosa.displayimport matplotlib.pyplot as pltimport numpy as npimport path as phimport pydub as pbimport speech_recognition as srimport warningsdef create_spectrogram(filename, name):    plt.interactive(False)    clip, sample_rate = librosa.load(filename, sr=None)    fig = plt.figure(figsize=[0.72, 0.72])    ax = fig.add_subplot(111)    ax.axes.get_xaxis().set_visible(False)    ax.axes.get_yaxis().set_visible(False)    ax.set_frame_on(False)    S = librosa.feature.melspectrogram(y=clip, sr=sample_rate)    librosa.display.specshow(librosa.power_to_db(S, ref=np.max))    filename = '../Images/Train/' + name + '.jpg'    plt.savefig(filename, dpi=400, bbox_inches='tight', pad_inches=0)    plt.close()    fig.clf()    plt.close(fig)    plt.close('all')    del filename, name, clip, sample_rate, fig, ax, S
Once we have converted each audio signal into an image representing the corresponding Spectrogram, we can load the dataset containing information on the labels of each audio. To properly link each audio file to its Sentiment, we create an ID column containing the name of the image file for that sample:
# Data Loadingtrain = pd.read_csv('../Data/MELD.Raw/train_sent_emo.csv', dtype=str)validation = pd.read_csv('../Data/MELD.Raw/dev_sent_emo.csv', dtype=str)test = pd.read_csv('../Data/MELD.Raw/test_sent_emo.csv', dtype=str)# Create mapping to identify audio filestrain["ID"] = 'dia'+train["Dialogue_ID"]+'_utt'+train["Utterance_ID"]+'.jpg'validation["ID"] = 'dia'+validation["Dialogue_ID"]+'_utt'+validation["Utterance_ID"]+'.jpg'test["ID"] = 'dia'+test["Dialogue_ID"]+'_utt'+test["Utterance_ID"]+'.jpg'
At the same time, we also need to take the text associated with an audio signal and process it using NLP techniques to transform it into a numeric vector so that the Neural Network can process it. Since we already have information on the text from the MELD dataset itself, we can go ahead with it. Otherwise, if the information was not available, to do this, we could use text conversion libraries like Google Cloud’s speech_recognition:
# Text Featurestokenizer = kt.Tokenizer(num_words=5000)tokenizer.fit_on_texts(train['Utterance'])vocab_size = len(tokenizer.word_index) + 1train_tokens = tokenizer.texts_to_sequences(train['Utterance'])text_features = pd.DataFrame(ks.pad_sequences(train_tokens, maxlen=200))validation_tokens = tokenizer.texts_to_sequences(validation['Utterance'])validation_features = pd.DataFrame(ks.pad_sequences(validation_tokens, maxlen=200))
We first tokenise the sentences spoken in each audio and then convert them into numeric vectors of length two hundred.
One of the trickiest aspects of putting together multi-media inputs, is the creation of a custom Data Generator. This structure is basically a function able to iteratively return to the model the next batch of input every time that it is called. Using Keras’ pre-made generator is relatively easy, but there is no implementation allowing you to merge together multiple inputs and make sure that both inputs are fed into the model side by side without errors.
The following code is general enough that it can be used in different settings, not just related to this example. In particular, it takes a folder location, where the images are located, and a “normal” dataset, having samples in the rows and features in the columns. It then iteratively provides the next samples of both images and text features in this case, both in batches of the same size:
# Data Pipelinedef train_generator(features, batch):    # Image Generator    train_generator = ki.ImageDataGenerator(        rescale=1. / 255.)    train_generator = train_generator.flow_from_dataframe(        dataframe=train,        directory="../Images/Train/",        x_col="ID",        y_col="Sentiment",        batch_size=batch,        seed=0,        shuffle=False,        class_mode="categorical",        target_size=(64, 64))    train_iterator = features.iterrows()    j = 0    i = 0    while True:        genX2 = pd.DataFrame(columns=features.columns)        while i < batch:            k,r = train_iterator.__next__()            r = pd.DataFrame([r], columns=genX2.columns)            genX2 = genX2.append(r)            j += 1            i += 1            if j == train.shape[0]:                X1i = train_generator.next()                train_generator = ki.ImageDataGenerator(                    rescale=1. / 255.)                train_generator = train_generator.flow_from_dataframe(                    dataframe=train,                    directory="../Images/Train/",                    x_col="ID",                    y_col="Sentiment",                    batch_size=batch,                    seed=0,                    shuffle=False,                    class_mode="categorical",                    target_size=(64, 64))                # Text Generator                train_iterator = features.iterrows()                i = 0                j=0                X2i = genX2                genX2 = pd.DataFrame(columns=features.columns)                yield [X1i[0], tf.convert_to_tensor(X2i.values, dtype=tf.float32)], X1i[1]        X1i = train_generator.next()        X2i = genX2        i = 0        yield [X1i[0], tf.convert_to_tensor(X2i.values, dtype=tf.float32)], X1i[1]
Finally, we can create the model having as input images (Spectrogram) and text (Transcriptions), and that processes them.
The inputs consist of images made of 64x64 pixels and 3 channels (RGB), and “normal” numeric features, of length 200, representing the encoding of the text:
# Model# Inputsimages = km.Input(shape=(64, 64, 3))features = km.Input(shape=(200, ))
The Image Classification branch is made of an initial VGG19 network and then a set of custom layers. By using VGG19, we can take advantage of the benefits of using Transfer Learning. In particular, since the model has been pre-trained on the ImageNet dataset, it already starts with coefficients initialised to values meaningful for image classification tasks. The output is then fed into a series of layers that can learn the specific characteristics of this type of images and then outputted to the next common layers:
# Transfer Learning Basesvgg19 = ka.VGG19(weights='imagenet', include_top=False)vgg19.trainable = False# Image Classification Branchx = vgg19(images)x = kl.GlobalAveragePooling2D()(x)x = kl.Dense(32, activation='relu')(x)x = kl.Dropout(rate=0.25)(x)x = km.Model(inputs=images, outputs=x)
The NLP Branch uses a Long Short-Term Memory (LSTM) layer, together with an Embedding layer to process the data. Dropout layers are also added to avoid the model overfishing, similarly to what done in the CNN Branch:
# Text Classification Branchy = kl.Embedding(vocab_size, EMBEDDING_LENGTH, input_length=200)(features)y = kl.SpatialDropout1D(0.25)(y)y = kl.LSTM(25, dropout=0.25, recurrent_dropout=0.25)(y)y = kl.Dropout(0.25)(y)y = km.Model(inputs=features, outputs=y)
We then can combine these two outputs and feed them into a series of Dense layers. A set of dense layers is able to capture information that is available only when both audio and text signals are combined, and is not identifiable from each input individually.
We also use an Adam optimizer with learning rate 0.0001, which is generally a great combination:
combined = kl.concatenate([x.output, y.output])z = kl.Dense(32, activation="relu")(combined)z = kl.Dropout(rate=0.25)(z)z = kl.Dense(32, activation="relu")(z)z = kl.Dropout(rate=0.25)(z)z = kl.Dense(3, activation="softmax")(z)model = km.Model(inputs=[x.input, y.input], outputs=z)model.compile(optimizer=ko.Adam(lr=0.0001), loss='categorical_crossentropy', metrics='accuracy')model.summary()
The model can then be trained using the Training and Validation Generators that we previously created by doing:
# HyperparametersEPOCHS = 13TRAIN_STEPS = np.floor(train.shape[0]/BATCH)VALIDATION_STEPS = np.floor(validation.shape[0]/BATCH)# Model Trainingmodel.fit_generator(generator=train_generator(text_features, BATCH),                    steps_per_epoch=TRAIN_STEPS,                    validation_data=validation_generator(validation_features, BATCH),                    validation_steps=VALIDATION_STEPS,                    epochs=EPOCHS)
Lastly, the model is evaluated on a set of data, for instance the Validation set alone, and then saved as a file to load when used in “Live” scenarios:
# Performance Evaluation# Validationmodel.evaluate_generator(generator=validation_generator(validation_features, BATCH))# Save the Model and Labelsmodel.save('Model.h5')
Overall, we built a system able to take multiple types of inputs (images, text, etc.), preprocess them and then feed them to a Neural Network consisting of a branch per input. Each branch individually processes its input for then converging into a common set of layers predicting the final output.
The specific steps are:
Data Loading
Preprocess Input Separately (Spectrogram, Tokenization)
Creating a Custom Data Generator
Building the Model Architecture
Model Training
Performance Evaluation
To read more articles like this, follow me on Twitter, LinkedIn or my Website.