What exactly is train_test_split doing to the data?

RaphaelL: <p>Is<br>
test_x, test2_x, test_y, test2_y = train_test_split(test_x, test_y, test_size=0.001, random_state=134515, stratify=test_y)<br>
doing anything (reshaping?) besides splitting the dataset? I have a trained model and I’m getting completely different accuracies (0.85 vs. 0.52!) if I make a forward pass with the whole dataset or with 99.99% of the dataset (after uncommenting the relevant line in the code below). Obviously the two images not included in test_x after the split don’t explain the difference so I 'm guessing the split command is something else but I can’t figure out what it is!</p>
<h1>loading dataset</h1>
<p>test = pd.read_csv(‘img_label_list04.csv’)<br>
test.head()</p>
<p><span class="hashtag">#---------</span></p>
<h1>loading test images</h1>
<p>test_img = []<br>
for img_name in tqdm(test[‘image_names’]):<br>
# defining the image path<br>
image_path = ‘d:/Pytorch/Data/leaves/set04/’ + img_name<br>
img = imread(image_path)<br>
# normalizing the pixel values<br>
img = img/255<br>
# resizing the image to (224,224,3)<br>
img = resize(img, output_shape=(224,224,3), mode=‘constant’, anti_aliasing=True)<br>
# converting the type of pixel to float 32<br>
img = img.astype(‘float32’)<br>
# appending the image into the list<br>
test_img.append(img)</p>
<h1>converting the list to numpy array</h1>
<p>test_x = np.array(test_img)<br>
test_x.shape</p>
<h1>defining the target</h1>
<p>test_y = train[‘infected_or_not’].values</p>
<p><span class="hashtag">#-----------------</span></p>
<h1>split the data sest (or not)</h1>
<h1>test_x, test2_x, test_y, test2_y = train_test_split(test_x, test_y, test_size=0.001, random_state=134515, stratify=test_y)</h1>
<h1>shape of test data</h1>
<p>test_x.shape, test_y.shape</p>
<h1>converting the test images into torch format</h1>
<p>test_x = test_x.reshape(len(test_x), 3, 224, 224)<br>
test_x  = torch.from_numpy(test_x)</p>
<p>test_y = test_y.astype(int)<br>
test_y = torch.from_numpy(test_y)</p>
<p>data_y = []<br>
label_y = []</p>
<p>inputs,labels = test_x, test_y</p>
<p>for i in tqdm(range(int(test_x.shape[0]/batch_size))):<br>
input_data = inputs[i*batch_size:(i+1)*batch_size]<br>
label_data = labels[i*batch_size:(i+1)*batch_size]<br>
input_data , label_data = Variable(input_data.cuda()),Variable(label_data.cuda())<br>
x = model.features(input_data)<br>
data_y.extend(x.data.cpu().numpy())<br>
label_y.extend(label_data.data.cpu().numpy())</p>
<p><span class="hashtag">#--------------------------</span></p>
<h1>converting the features into torch format</h1>
<p>x_test  = torch.from_numpy(np.array(data_y))<br>
x_test = x_test.view(x_test.size(0), -1)<br>
y_test  = torch.from_numpy(np.array(label_y))</p>
<p><span class="hashtag">#--------------------------</span></p>
<h1>prediction for test set</h1>
<p>prediction_test = []<br>
target_test = []<br>
permutation = torch.randperm(x_test.size()[0])<br>
for i in tqdm(range(0,x_test.size()[0], batch_size)):<br>
indices = permutation[i:i+batch_size]<br>
batch_x, batch_y = x_test[indices], y_test[indices]</p>
<pre><code>if torch.cuda.is_available():
    batch_x, batch_y = batch_x.cuda(), batch_y.cuda()

with torch.no_grad():
    output = model.classifier(batch_x.cuda())

softmax = torch.exp(output).cpu()
prob = list(softmax.numpy())
predictions = np.argmax(prob, axis=1)
prediction_test.append(predictions)
target_test.append(batch_y)
</code></pre>
<h1>test accuracy</h1>
<p>accuracy_test = []<br>
for i in range(len(prediction_test)):<br>
np_target_test=target_test[i].cpu().numpy()<br>
accuracy_test.append(accuracy_score(np_target_test,prediction_test[i]))<br>
accuracy.append(accuracy_score(np_target_test,prediction_test[i]))</p>
<p>print(‘test accuracy: \t’, np.average(accuracy_test))</p>

ptrblck: <p>The method just splits the data using your arguments as described in <a href="https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html">the docs</a>.</p>
<p>I assume you want to permute the dimensions in this line of code:</p>
<pre><code class="lang-python">test_x = test_x.reshape(len(test_x), 3, 224, 224)
</code></pre>
<p>which won’t work as expected, since <code>reshape</code> will interleave the data. Use <code>transpose</code> in numpy or <code>permute</code> in PyTorch instead to swap the dimensions.</p>

RaphaelL: <p>Thanks for your suggestion but unfortunately this doesn’t solve the problem. To make things easier, the code pasted below includes 2 parts (TEST1 and TEST 2), where the second part was created simply by pasting the first part and deleting the “train_test_split” line. The accuracy at the end of part 1 is 0.93 and the accuracy at the end of part 2 is 0.70!</p>
<h1>TEST 1 after splitting the dataset and using only 99% of the images</h1>
<p><span class="hashtag">#--------------------------------------------------------------------</span></p>
<h1>loading dataset</h1>
<p>test = []<br>
test = pd.read_csv(‘img_label_list04.csv’)<br>
test.head()</p>
<p><span class="hashtag">#---------</span></p>
<h1>loading test images</h1>
<p>test_img = []<br>
for img_name in tqdm(test[‘image_names’]):<br>
# defining the image path<br>
image_path = ‘d:/Pytorch/Data/leaves/set04/’ + img_name<br>
# reading the image<br>
img = imread(image_path)<br>
# normalizing the pixel values<br>
img = img/255<br>
# resizing the image to (224,224,3)<br>
img = resize(img, output_shape=(224,224,3), mode=‘constant’, anti_aliasing=True)<br>
# converting the type of pixel to float 32<br>
img = img.astype(‘float32’)<br>
# appending the image into the list<br>
test_img.append(img)</p>
<h1>converting the list to numpy array</h1>
<p>test_x = np.array(test_img)<br>
test_x.shape</p>
<h1>defining the target</h1>
<p>test_y = test[‘infected_or_not’].values</p>
<p><span class="hashtag">#-----------------</span></p>
<h1>split the data set</h1>
<p>test_x, test2_x, test_y, test2_y = train_test_split(test_x, test_y, test_size=0.001, random_state=134515, stratify=test_y)</p>
<h1>shape of test data</h1>
<p>test_x.shape, test_y.shape</p>
<h1>converting the test images into torch format</h1>
<p>test_x = test_x.transpose(0,3,1,2)<br>
test_x  = torch.from_numpy(test_x)</p>
<p>test_y = test_y.astype(int)<br>
test_y = torch.from_numpy(test_y)</p>
<p>data_test = []<br>
label_test = []</p>
<p>inputs,labels = test_x, test_y</p>
<p>for i in tqdm(range(int(test_x.shape[0]/batch_size))):<br>
input_data = inputs[i*batch_size:(i+1)*batch_size]<br>
label_data = labels[i*batch_size:(i+1)*batch_size]<br>
input_data , label_data = Variable(input_data.cuda()),Variable(label_data.cuda())<br>
x = model.features(input_data)<br>
data_test.extend(x.data.cpu().numpy())<br>
label_test.extend(label_data.data.cpu().numpy())</p>
<p><span class="hashtag">#--------------------------</span></p>
<h1>converting the features into torch format</h1>
<p>x_test  = torch.from_numpy(np.array(data_test))<br>
x_test = x_test.view(x_test.size(0), -1)<br>
y_test  = torch.from_numpy(np.array(label_test))</p>
<p><span class="hashtag">#--------------------------</span></p>
<h1>prediction for test set</h1>
<p>prediction_test = []<br>
target_test = []</p>
<h1>permutation = torch.randperm(x_test.size()[0])</h1>
<p>for i in tqdm(range(0,x_test.size()[0], batch_size)):<br>
# indices = permutation[i:i+batch_size]<br>
indices = range(i,i+batch_size)<br>
batch_x, batch_y = x_test[indices], y_test[indices]</p>
<pre><code>if torch.cuda.is_available():
    batch_x, batch_y = batch_x.cuda(), batch_y.cuda()

with torch.no_grad():
    output = model.classifier(batch_x.cuda())

softmax = torch.exp(output).cpu()
prob = list(softmax.numpy())
predictions = np.argmax(prob, axis=1)
prediction_test.append(predictions)
target_test.append(batch_y)
</code></pre>
<h1>test accuracy</h1>
<p>accuracy_test = []<br>
for i in range(len(prediction_test)):<br>
np_target_test=target_test[i].cpu().numpy()<br>
accuracy_test.append(accuracy_score(np_target_test,prediction_test[i]))</p>
<p>print(‘test accuracy 1: \t’, np.average(accuracy_test))</p>
<h1>TEST 2 using all the images</h1>
<h1>-------------------------------------------------</h1>
<p>test = []<br>
test = pd.read_csv(‘img_label_list04.csv’)<br>
test.head()</p>
<p><span class="hashtag">#---------</span></p>
<h1>loading test images</h1>
<p>test_img = []<br>
for img_name in tqdm(test[‘image_names’]):<br>
# defining the image path<br>
image_path = ‘d:/Pytorch/Data/leaves/set04/’ + img_name<br>
# reading the image<br>
img = imread(image_path)<br>
# normalizing the pixel values<br>
img = img/255<br>
# resizing the image to (224,224,3)<br>
img = resize(img, output_shape=(224,224,3), mode=‘constant’, anti_aliasing=True)<br>
# converting the type of pixel to float 32<br>
img = img.astype(‘float32’)<br>
# appending the image into the list<br>
test_img.append(img)</p>
<h1>converting the list to numpy array</h1>
<p>test_x = np.array(test_img)<br>
test_x.shape</p>
<h1>defining the target</h1>
<p>test_y = test[‘infected_or_not’].values</p>
<p><span class="hashtag">#-----------------</span></p>
<h1>shape of test data</h1>
<p>test_x.shape, test_y.shape</p>
<h1>converting the test images into torch format</h1>
<p>test_x = test_x.transpose(0,3,1,2)<br>
test_x  = torch.from_numpy(test_x)</p>
<p>test_y = test_y.astype(int)<br>
test_y = torch.from_numpy(test_y)</p>
<p>data_test = []<br>
label_test = []</p>
<p>inputs,labels = test_x, test_y</p>
<p>for i in tqdm(range(int(test_x.shape[0]/batch_size))):<br>
input_data = inputs[i*batch_size:(i+1)*batch_size]<br>
label_data = labels[i*batch_size:(i+1)*batch_size]<br>
input_data , label_data = Variable(input_data.cuda()),Variable(label_data.cuda())<br>
x = model.features(input_data)<br>
data_test.extend(x.data.cpu().numpy())<br>
label_test.extend(label_data.data.cpu().numpy())</p>
<p><span class="hashtag">#--------------------------</span></p>
<h1>converting the features into torch format</h1>
<p>x_test  = torch.from_numpy(np.array(data_test))<br>
x_test = x_test.view(x_test.size(0), -1)<br>
y_test  = torch.from_numpy(np.array(label_test))</p>
<p><span class="hashtag">#--------------------------</span></p>
<h1>prediction for test set</h1>
<p>prediction_test = []<br>
target_test = []</p>
<h1>permutation = torch.randperm(x_test.size()[0])</h1>
<p>for i in tqdm(range(0,x_test.size()[0], batch_size)):<br>
# indices = permutation[i:i+batch_size]<br>
indices = range(i,i+batch_size)<br>
batch_x, batch_y = x_test[indices], y_test[indices]</p>
<pre><code>if torch.cuda.is_available():
    batch_x, batch_y = batch_x.cuda(), batch_y.cuda()

with torch.no_grad():
    output = model.classifier(batch_x.cuda())

softmax = torch.exp(output).cpu()
prob = list(softmax.numpy())
predictions = np.argmax(prob, axis=1)
prediction_test.append(predictions)
target_test.append(batch_y)
</code></pre>
<h1>test accuracy</h1>
<p>accuracy_test = []<br>
for i in range(len(prediction_test)):<br>
np_target_test=target_test[i].cpu().numpy()<br>
accuracy_test.append(accuracy_score(np_target_test,prediction_test[i]))</p>
<p>print(‘test accuracy 2: \t’, np.average(accuracy_test))</p>

saba: <p>Dear Ptrblck ,</p>
<p>I applied:</p>
<pre><code class="lang-auto">train_test_split(WholeData1.numpy(), 
wholetargetArray,train_size=0.7,test_size=0.3,stratify=wholetargetArray)```


the wholetargetArray is the whole balanced data 4000positive and 4000 negative, and wholetargetArray is the whole 8000 target regarding the dataset.
the number of portion is good , the ValidLabel, and TrainLabel are binary labels  but the ValidationData1 , and TrainData1 are zero!</code></pre>

ptrblck: <p>I’m not sure, why the data split should be zero.<br>
Anyway, you could only pass the target array to <code>stratify</code> and use <code>np.arange(len(target))</code> as the “data” input, to get the split indices as the output.</p>