Custumized dataset from JSON file

username: Oussama_Bouldjedri
cooked: <p>Iam working using JSON files, the structure of the file is :<br>
{“data”: [{<br>
“ID”:…,<br>
“SensorName”:… ,<br>
“DigitalValue”: …,<br>
“AnalogValue”:… ,<br>
“SchematicName”:…,<br>
“Pin”:…,<br>
“Enable”:…,<br>
“TimeStamp”:…,</p>
<p>}]}</p>
<p>each file is made of many structure like the one above I was wondering how to write a custumized dataset for pythorch using this I get errors:</p>
<p>class JsonDataset(IterableDataset):<br>
def <strong>init</strong>(self, files):<br>
self.files = files<br>
self.label=label<br>
print('self files ',self.files)<br>
input()</p>
<pre><code>def __len__(self):
    
    return ()
         
def __getitem__(self, idx):
    print('get item hello')
    input()
    data = []
    for k, v in idx['data'][0].items():
        print('v is ',v)
        print()
        print('k is ',k)
        input()
        if type(v) == str:
            # convert it into int or float
            v=int(v)
            data.append(v)
            # and append into data
            
            
        else:
            
            data.append(v)
           
            
            
    torch_data = torch.Tensor(data)
    label=Y
    input()
</code></pre>
<p>thanks</p>

username: nivek
cooked: <p>Hi, based on your code snippet, it doesn’t seem like your <code>__getitem__</code> method is returning anything. Maybe try adding <code>return torch_data</code> at the end?</p>
<p>If you include your error message here, it will be helpful as well.</p>

username: Oussama_Bouldjedri
cooked: <p>thanks , I think my own issue is that I am not able to iterate my json file for which the structure is:</p>
<p>{“serial_number”:"…",“timestamp”:…,“data”:[…]}</p>
<p>do you have an idea about how to iterate this specefic type of structure ?</p>
<p>PS: i have many files so the structure of the second type is different compared to the one posted already</p>

username: nivek
cooked: <p>Depending JSON file structure and which level you would like to iterate through, it will be different.</p>
<p>If the JSON file is a list of entries, and each entry is a dict like this <code>{“serial_number”:"…",“timestamp”:…,“data”:[…]}</code>, then:</p>
<pre><code class="lang-python">dictionaries = json.load(f)
for row in dictionaries:
    data = row['data']
</code></pre>
<p>If the JSON file is dict with keys such as “serial_number”, “timestamp”, “data” (i.e. <code>{“serial_number”:"…",“timestamp”:…,“data”:[…]}</code>) , then:</p>
<pre><code class="lang-python">d = json.load(f)
serial_number = d['serial_number']
timestamp = d['timestamp']
data = d['data']
for point in data:
    # You can iterate through the data here, or you can just convert it to a Tensor
</code></pre>

username: Oussama_Bouldjedri
cooked: <aside class="quote no-group" data-username="nivek" data-post="4" data-topic="142078">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nivek/48/39672_2.png" class="avatar"> nivek:</div>
<blockquote>
<pre><code class="lang-auto">d = json.load(f)
serial_number = d['serial_number']
timestamp = d['timestamp']
data = d['data']
for point in data:
</code></pre>
</blockquote>
</aside>
<p>both lines of json.load(f) get me to this error:<br>
raise JSONDecodeError(“Extra data”, s, end)</p>
<p>JSONDecodeError: Extra data</p>

username: nivek
cooked: <p>It is hard to debug without being able to see the file that you are trying to load, but the answers here should help: <a href="https://stackoverflow.com/questions/21058935/python-json-loads-shows-valueerror-extra-data" class="inline-onebox" rel="noopener nofollow ugc">Python json.loads shows ValueError: Extra data - Stack Overflow</a></p>

username: Oussama_Bouldjedri
cooked: <p>it seems that my file is not well formatted some how linked to the {}, thnaks for the help.</p>