How to Preprocess Character Level Text with Keras | by Xu LIANG | Towards Data Science

You can find the notebook here
The goal of this introduction is to let you know how to use Keras to preprocess text in character level. The rest of the article is organized as follows.
Load data
Tokenizer
Change vocabulary
Character to index
Padding
Get Labels
First, we use pandas to load the training data.
Combining column 1 and column 2 as one text.
Saving the column 1 to texts and convert all sentence to lower case.
When initializing the Tokenizer, there are only two parameters important.
char_level=True: this can tell tk.texts_to_sequences() to process sentence in char level.
oov_token='UNK': this will add a UNK token in the vocabulary. We can call it by tk.oov_token.
After call tk.fit_on_texts(texts), tk class will contain the neccery information about the training data. We can call tk.word_index to see the dictionary. Here the index of UNK is word_count+1.
This is the character dictionary learned from the training data. But if we already have a character list, we have to change the tk_word_index.
See I already have a character list call alphabet, we build a char_dict based on alphabet.
We will assign a new index to UNK.
After we get the right vocabulary, we can represent all texts by using character index.
This step is very simple, tk.texts_to_sequences() will do this conversion automatically for us.
We can see the string representation is replaced by index representation. We list top 5 sentence length.
Because text have different length, we have to make all text as the same length, so the CNN can handle the batch data.
Here we set the max sentence length as 1014. If the length of a text is smaller than 1014, the rest of the part will be padded as 0. If the length of a text is bigger than 1014, the part longer than 1014 will be truncated. So all texts will maintain the same length.
Finally, we convert the list to Numpy array.
First, we assign column 0 in train_df to a class_list, this 1-dimensional list contains all labels for each text. But our task is a multiclass task, so we have to convert it to a 2-dimensional array. Here we could use the to_categorical method in Keras
As for the test dataset, we just need to do the same process again.
For easy use, I sum all code together.
# write all code in one cell#========================Load data=========================import numpy as npimport pandas as pdtrain_data_source = '../data/ag_news_csv/train.csv'test_data_source = '../data/ag_news_csv/test.csv'train_df = pd.read_csv(train_data_source, header=None)test_df = pd.read_csv(test_data_source, header=None)# concatenate column 1 and column 2 as one textfor df in [train_df, test_df]:    df[1] = df[1] + df[2]    df = df.drop([2], axis=1)    # convert string to lower case train_texts = train_df[1].values train_texts = [s.lower() for s in train_texts]test_texts = test_df[1].values test_texts = [s.lower() for s in test_texts]#=======================Convert string to index================from keras.preprocessing.text import Tokenizerfrom keras.preprocessing.sequence import pad_sequences# Tokenizertk = Tokenizer(num_words=None, char_level=True, oov_token='UNK')tk.fit_on_texts(train_texts)# If we already have a character list, then replace the tk.word_index# If not, just skip below part#-----------------------Skip part start--------------------------# construct a new vocabulary alphabet="abcdefghijklmnopqrstuvwxyz0123456789-,;.!?:'\"/\\|_@#$%^&*~`+-=<>()[]{}"char_dict = {}for i, char in enumerate(alphabet):    char_dict[char] = i + 1    # Use char_dict to replace the tk.word_indextk.word_index = char_dict # Add 'UNK' to the vocabulary tk.word_index[tk.oov_token] = max(char_dict.values()) + 1#-----------------------Skip part end----------------------------# Convert string to index train_sequences = tk.texts_to_sequences(train_texts)test_texts = tk.texts_to_sequences(test_texts)# Paddingtrain_data = pad_sequences(train_sequences, maxlen=1014, padding='post')test_data = pad_sequences(test_texts, maxlen=1014, padding='post')# Convert to numpy arraytrain_data = np.array(train_data)test_data = np.array(test_data)#=======================Get classes================train_classes = train_df[0].valuestrain_class_list = [x-1 for x in train_classes]test_classes = test_df[0].valuestest_class_list = [x-1 for x in test_classes]from keras.utils import to_categoricaltrain_classes = to_categorical(train_class_list)test_classes = to_categorical(test_class_list)
Check out my other posts on Medium with a categorized view!GitHub: BrambleXuLinkedIn: Xu LiangBlog: BrambleXu