Create Json Dataset

Alphonsito25: <p>Hi, I am trying to create a class in order to read and create a json dataset for use in a CNN.<br>
The json files are like this (pose keypoints from images):</p>
<pre><code>  {
  "0": {
    "PoseKeypoints": [
        [
            2529.287109375,
            1424.733642578125,
            0.9513001441955566
        ],
        [
            2574.495849609375,
            1384.9786376953125,
            0.9392595291137695
        ],
        [ ...
</code></pre>
<p>I have to store this keypoints into a pytorch tensor.<br>
My idea is to use iterable objects in the class, but I don’t know how to do it.<br>
Thank you!!</p>

Alphonsito25: <p>I have written this model</p>
<pre><code>data = []

class json_dataset(Dataset):
    def __init__(self, root_dir):
        self.root_dir= root_dir
    def __getitem__(self,index):
        for file in os.listdir(self.root_dir):
            if file.endswith('json'):
                json_path = os.path.join(self.root_dir, file)
                
                #json_data = pd.read_json(json_path, lines=True)
                json_data = json.load(open(json_path))
                
                for keypoints in json_data.items():
                    valores = keypoints['PoseKeypoints']
                    keypoints_normalized.append(valores)
                
                data.append(json_data)
                data = torch.FloatTensor(data)
            return(data)
</code></pre>
<p>What do you think?</p>

ptrblck: <p>The general loading looks alright (replace <code>torch.FloatTensor</code> with e.g. <code>torch.from_numpy</code> or <code>torch.tensor</code>), but the for loop looks wrong.<br>
In the <code>__getitem__</code> method you would use the <code>index</code> to load a single sample while it seems you are trying to iterate all json files and return the very first one all all indices.</p>

Alphonsito25: <p>Now I have this:</p>
<pre><code>class json_dataset(Dataset):
    keypoints = []
    def __init__(self, root_dir):
    	self.root_dir= root_dir
    def __str__(self):	
    	return str(data) #???
    def __getitem__(self,index):
    	for file in os.listdir(self.root_dir):
    		if file.endswith('json'):
        		json_path = os.path.join(self.root_dir, file)

        		json_data = json.load(open(json_path))

        		for k in json_data["0"]["PoseKeypoints"]:
        			keypoints.append(k)

        		data.append(keypoints)
        		data = torch.Tensor(data)
    	return(data)
</code></pre>
<p>How do I do what you tell me about the index?</p>
<p>Also I want to check if the return “data” is correct, but when I print it returns me an empty tensor. I checked out of a class and it should me return well.<br>
The <strong>str</strong> function is for this.</p>
<p>Thank you!</p>

ptrblck: <aside class="quote no-group" data-username="Alphonsito25" data-post="4" data-topic="156826">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/d6d6ee/48.png" class="avatar"> Alphonsito25:</div>
<blockquote>
<p>How do I do what you tell me about the index?</p>
</blockquote>
</aside>
<p>In the common use case you have a specific number of samples in the <code>Dataset</code> and this number of samples is returned in the <code>Dataset.__len__</code> function. In the <code>__getitem__</code> method you are using the <code>index</code> (which has values in the range <code>[0, len(dataset)-1]</code>) to load a single sample for this <code>index</code>.<br>
E.g. if each sample is stored in a separate <code>json</code> file in the <code>self.root_dir</code> you could load the corresponding file using the <code>index</code> instead of iterating all files.</p>
<aside class="quote no-group" data-username="Alphonsito25" data-post="4" data-topic="156826">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/d6d6ee/48.png" class="avatar"> Alphonsito25:</div>
<blockquote>
<p>Also I want to check if the return “data” is correct, but when I print it returns me an empty tensor.</p>
</blockquote>
</aside>
<p>Add <code>print</code> statements to the <code>__getitem__</code> method and check which objects are valid and where the tensor becomes an empty one.</p>
<aside class="quote no-group" data-username="Alphonsito25" data-post="4" data-topic="156826">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/d6d6ee/48.png" class="avatar"> Alphonsito25:</div>
<blockquote>
<p>The <strong>str</strong> function is for this.</p>
</blockquote>
</aside>
<p>The <code>__str__</code> function uses <code>data</code>, which is undefined or globally defined, so check what is being printed there.</p>

Alphonsito25: <p>Ok, I made some changes:</p>
<pre><code>class json_dataset(Dataset):
    def __init__(self, csv_file, root_dir):
    	self.annotations= pd.read_csv(csv_file)
    	self.root_dir= root_dir
    	
    def __len__(self):
    	return len(self.annotations)
    	
    def __getitem__(self,index):
    	json_path= os.path.join(self.root_dir, self.annotations.iloc[index, 0])
    	json_data = json.load(open(json_path))
        keypoints = []
	
    	if "0" in json_data:
    		for k in json_data["0"]["PoseKeypoints"]:
    			keypoints.append(k)     				
    	data.append(keypoints)
    	data = torch.Tensor(data)
    	return(data)
</code></pre>
<p>The csv file has the names of the json files. Now it works, thank you!<br>
But I have a question: How the index works so that in training you can access all the images? I need that the dataset loads sorted with the filenames.</p>