WeightedRandomSampler sampler not working properly

username: Sharut_Gupta
cooked: <p>I am using the Weighted random sampler function of PyTorch to sample my classes equally, But while checking the samples of each class in a batch, it seems to sample randomly.</p>
<p>here is a snippet of my code.</p>
<p>def cal_sample_weight(files):<br>
print("file length ",len(files))<br>
labels = [int(f[-5])-1 for f in files]<br>
class_count = [labels.count© for c in np.unique(labels)]<br>
print("class count is ",class_count)<br>
# weights_class = [(1.0/len(np.unique(labels)))/(cc) for cc in class_count]<br>
weights_class = [1.0/(cc) for cc in class_count]<br>
print("Weight class is ", weights_class)<br>
weight_array = np.array([weights_class[l] for l in labels])<br>
print(weight_array.shape)<br>
return torch.from_numpy(weight_array)</p>
<p>np.random.seed(0)<br>
random.shuffle(Train_files)<br>
random.shuffle(Val_files)</p>
<p>print([(int(image_path[-5])-1) for image_path in Train_files[0:32]])<br>
Train_labels = [(int(image_path[-5])-1) for image_path in Train_files]<br>
Val_labels = [(int(image_path[-5])-1) for image_path in Val_files]</p>
<h1></h1>
<p>Train_sample_weight = cal_sample_weight(Train_files)<br>
Val_sample_weight = cal_sample_weight(Val_files)</p>
<p>training_set = Dataset(list_IDs=Train_files,labels=Train_labels, sample_weight=Train_sample_weight, shuffle=True, augment=True)<br>
print("length of train is ", len(training_set))</p>
<p>sampler_train = torch.utils.data.sampler.WeightedRandomSampler(weights = Train_sample_weight, num_samples=len(training_set))<br>
trainloader = DataLoader(torch.utils.data.Subset(training_set,Train_labels), batch_size=batch_size, sampler=sampler_train,shuffle=False, num_workers=24, pin_memory=True)</p>

username: ptrblck
cooked: <p>I can’t find any obvious errors in the code.<br>
Could you post the current class distribution you are seeing in each batch?</p>
<p>I assume <code>Train_labels</code> contains the labels for the complete <code>training_set</code>?<br>
If so you don’t need to wrap it into a <code>Subset</code>.</p>
<p>PS: You can post code snippets by wrapping them into three backticks ```, which would make debugging easier. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: Sharut_Gupta
cooked: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/8/98e0038b3e69dbd3a482a7cd01a57f259cb32dfc.png" alt="image" data-base62-sha1="lOowqeEZ52diYcHThKo0obqwlCQ" width="477" height="400"><br>
This is the distribution most of the times. Distribution is pretty random. If I remove the subset part and the labels part, then none of the class has samples as 0 in the batch, but the sampling contains majority from class 2,3 and minority from 1,4 which should not be the case.<br>
Its sampling is random. When I run the code, sometimes it samples really badly, sometimes its better. Why is this so?<br>
Yes train labels contains labels for complete set.</p>
<pre><code class="lang-auto">class Dataset(data.Dataset):
	def __init__(self, list_IDs, labels, sample_weight, augment=True, shuffle=True):
		#'Initialization'
		self.list_IDs = list_IDs
		self.labels=labels
		self.augment = augment
		self.shuffle = shuffle
		self.sample_weight = sample_weight
		# self.on_epoch_end()
# 
	def __getitem__(self, index):
		np.random.seed(index)
		# image_path = np.random.choice(a=self.list_IDs)
		image_path = self.list_IDs[index]
		my_path=image_path.replace("/home/ken.chang@ccds.io/mnt","/data" )
		image = np.load(my_path)
		if self.augment:
			image = self.__preprocess_input(image)
		label = (int(image_path[-5])-1)
		image = TF.to_tensor(image).float()
		label = torch.from_numpy(np.array(label))
		return image, label


	def __len__(self):
		return int(len(self.list_IDs))

	def on_epoch_end(self):
		print("hello_epoch")
		if self.shuffle == True:
			c = list(zip(self.list_IDs,self.sample_weight, self.labels))
			random.shuffle(c)
			self.list_IDs, self.sample_weight, self.labels = zip(*c)

	def __preprocess_input(self, image):
		if np.random.rand(1)[0]&gt;.5:
			image = np.fliplr(image)
		if np.random.rand(1)[0]&gt;.5:
			image = np.flipud(image)
		image = rotate(image, angle=np.random.rand(1)[0]*45, mode='nearest', reshape=False)
		return image

#draw from each sample with equal class probability
def cal_sample_weight(files):
	print("file length ",len(files))
	labels = [int(f[-5])-1 for f in files]
	class_count = [labels.count(c) for c in np.unique(labels)]
	print("class count is ",class_count)
	weights_class = [((1.0/len(np.unique(labels)))/(cc)) for cc in class_count]
	# weights_class = [1.0/(cc) for cc in class_count]
	print("Weight class is ", weights_class)
	weight_array = np.array([weights_class[l] for l in labels])
	return torch.from_numpy(weight_array)
	# return weight_array


os.chdir('/data/2015P002510/MammoDensity/scripts_preprocessing/splits')
with open("Train_images_DMIST2.txt", "rb") as fp:
	Train_files1 = pickle.load(fp)
with open("Train_images_DMIST3.txt", "rb") as fp:
	Train_files2 = pickle.load(fp)
with open("Train_images_DMIST4.txt", "rb") as fp:
	Train_files3 = pickle.load(fp)
with open("Train_images_MGH.txt", "rb") as fp:
	Train_files4 = pickle.load(fp)
Train_files = Train_files1+Train_files2+Train_files3+Train_files4
#
with open("Val_images_DMIST2.txt", "rb") as fp:
	Val_files1 = pickle.load(fp)
with open("Val_images_DMIST3.txt", "rb") as fp:
	Val_files2 = pickle.load(fp)
with open("Val_images_DMIST4.txt", "rb") as fp:
	Val_files3 = pickle.load(fp)
with open("Val_images_MGH.txt", "rb") as fp:
	Val_files4 = pickle.load(fp)
Val_files = Val_files1+Val_files2+Val_files3+Val_files4

print([(int(image_path[-5])-1) for image_path in Train_files[0:32]])

random.shuffle(Train_files)
random.shuffle(Val_files)


print([(int(image_path[-5])-1) for image_path in Train_files[0:32]])
Train_labels = [(int(image_path[-5])-1) for image_path in Train_files]
Val_labels = [(int(image_path[-5])-1) for image_path in Val_files]

#
Train_sample_weight = cal_sample_weight(Train_files)
print(Train_sample_weight[0:32])
Val_sample_weight = cal_sample_weight(Val_files)


# LOADING DATASET 
training_set = Dataset(list_IDs=Train_files,labels=Train_labels, sample_weight=Train_sample_weight, shuffle=True, augment=True)
print("length of train is ", len(training_set))

sampler_train = torch.utils.data.sampler.WeightedRandomSampler(weights = Train_sample_weight, num_samples=len(training_set))
trainloader = DataLoader(torch.utils.data.Subset(training_set,Train_labels), batch_size=batch_size, sampler=sampler_train,num_workers=24, pin_memory=True)
# trainloader = DataLoader(dataset=training_set, batch_size=batch_size, sampler=sampler_train,shuffle=False, num_workers=24, pin_memory=True)


test_set = Dataset(Val_files,labels=Val_labels, sample_weight=Val_sample_weight, augment=False)
print("lenght of val is ", len(test_set))
sampler_test = torch.utils.data.sampler.WeightedRandomSampler(weights = Val_sample_weight, num_samples=len(test_set))
testloader = DataLoader(dataset=test_set, batch_size=batch_size, sampler=sampler_test, num_workers=24, pin_memory=True)</code></pre>

username: ptrblck
cooked: <p>Could you provide some information about your current data distribution, i.e. how many instances do you have from each class?</p>

username: Sharut_Gupta
cooked: <p>Hey, I figured this out. Seems that even though I am zipping my weights with the samples while shuffling at the epoch end, the weights with the sampler don’t get shuffled. The weighted random sampler just allocated the probabilities to all the samples once and has no effect of my custom shuffling.</p>
<p>My accuracies are still 1-2% off as compared to my Keras code and I think the reason is ‘no-shuffle’ after each epoch. If I set shuffle=True in the data loader, it throws the error that sampler and shuffle are mutually exclusive options.<br>
<strong>Is there a way to shuffle properly with a weighted random sampler?</strong></p>

username: ptrblck
cooked: <p>The weights are used to sample the sample, which already samples “randomly” using the weights as probabilities.<br>
You cannot shuffle the dataset again.</p>

username: Sharut_Gupta
cooked: <p>Sorry, what is the meaning of “Randomly” here. Also, does the random seed being used in the WeightedRandomSampler to choose the batch getting changed after each epoch? since otherwise it will be choosing same batch again and again</p>

username: ptrblck
cooked: <p>Random sampling is done using some kind of probability distribution.<br>
If you are shuffling the data, it could be seen as sampling from a uniform distribution without replacement.<br>
Since you are providing the weights to sample each data sample, how should shuffling work in this case?</p>

username: Sharut_Gupta
cooked: <p>Thanks for your response.</p>
<p>I meant that since each sample has a weight, what if we can zipping the sample and the weight, shuffle them randomly, further unzip them and now use the sampler to draw the samples</p>

username: ptrblck
cooked: <p>That wouldn’t change anything, as the weight is still assigned to the same sample.<br>
Inside the <code>WeightedRandomSampler</code> <a href="https://github.com/pytorch/pytorch/blob/0bde610c14b92d351b968a0228df29e92442b1cc/torch/utils/data/sampler.py#L160">this line of code</a> will return the samples.<br>
If the order of the weights and samples is shuffled it won’t make a difference, as <code>torch.multinomial</code> will still use the same weight for the corresponding sample.</p>

username: Sharut_Gupta
cooked: <p>Okay. Thanks a lot for your response.</p>