Application of map(), then save_to_disk() leaves unchanged dataset?

username: rbelew
cooked: <p>I must be misunderstanding something: if I do this:</p>
<pre><code class="lang-auto">ds = datasets.load_dataset(csvDir,data_files=dsFiles,download_mode="force_redownload")
ds.map(preprocessOne)
ds.save_to_disk(mapDir)
</code></pre>
<p>and then reload via <code>ds2 = datasets.load_from_disk(mapDir)</code>, the new <code>ds2</code> does not show the benefits of the <code>preprocessOne</code> mapping?!  what would that be?</p>

username: anantguptadbl
cooked: <aside class="quote no-group" data-username="rbelew" data-post="1" data-topic="205968">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rbelew/48/66503_2.png" class="avatar"> rbelew:</div>
<blockquote>
<p><code>preprocessOne</code></p>
</blockquote>
</aside>
<p>Let us take  a look at what ds actually produces<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/9/29b36180677f626d319c6f13814cb6f27347cd9b.png" data-download-href="https://discuss.pytorch.org/uploads/default/29b36180677f626d319c6f13814cb6f27347cd9b" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/9/29b36180677f626d319c6f13814cb6f27347cd9b_2_690x220.png" alt="image" data-base62-sha1="5WTTFZoFk4mPbylK44zj1vF16NZ" width="690" height="220" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/9/29b36180677f626d319c6f13814cb6f27347cd9b_2_690x220.png, https://discuss.pytorch.org/uploads/default/original/3X/2/9/29b36180677f626d319c6f13814cb6f27347cd9b.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/2/9/29b36180677f626d319c6f13814cb6f27347cd9b.png 2x" data-dominant-color="F0F2F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">870×278 31.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>This is the important part</p>
<pre><code class="lang-auto">&lt;class 'datasets.formatting.formatting.LazyRow'&gt;
</code></pre>
<p>You can get more info on this here</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://huggingface.co/docs/datasets/v1.6.2/exploring.html">
  <header class="source">
      <img src="https://huggingface.co/docs/datasets/v1.6.2/_static/favicon.ico" class="site-icon" width="256" height="256">

      <a href="https://huggingface.co/docs/datasets/v1.6.2/exploring.html" target="_blank" rel="noopener nofollow ugc">huggingface.co</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/372;"><img src="https://huggingface.co/front/thumbnails/docs/datasets.png" class="thumbnail" width="690" height="372"></div>

<h3><a href="https://huggingface.co/docs/datasets/v1.6.2/exploring.html" target="_blank" rel="noopener nofollow ugc">What’s in the Dataset object</a></h3>

  <p>The datasets.Dataset object that you get when you execute for instance the following commands:&gt;&gt;&gt; from datasets import load_dataset &gt;&gt;&gt; dataset = load_datase...</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>The root cause is that you need to leverage the return value in map to update the data</p>
<pre><code class="lang-auto">import pandas as pd
from datasets import load_dataset

data = pd.DataFrame([[1,2,3], [4,5,6]], columns=['col1', 'col2', 'col3'])
data.to_csv(r"data/randomData.csv", index=False)
ds = load_dataset("data",data_files="randomData.csv",download_mode="force_redownload")

def preprocessOne(rowVal):
    rowVal['col1']=99
    return rowVal
    
dsUpdated = ds.map(lambda curRow: preprocessOne(curRow))
print(dsUpdated['train'][0])
dsUpdated.save_to_disk("data")
</code></pre>

username: rbelew
cooked: <p>thanks for your comments <a class="mention" href="/u/anantguptadbl">@anantguptadbl</a> but I’m afraid I’m missing your point.  The doc you point is flagged as legacy, with current doc being this: <a href="https://huggingface.co/docs/datasets/v2.20.0/access" class="inline-onebox" rel="noopener nofollow ugc">Know your dataset</a>.</p>
<p>and I am “leveraging” the return value. my preprocess function is for multiple choice examples (ala swag, cf. <a href="https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/multiple_choice.ipynb" class="inline-onebox" rel="noopener nofollow ugc">Google Colab</a>)</p>
<pre><code class="lang-auto">def preprocessOne(example):
	first_sentences = [example[Context_name] for i in range(NMultChoice)]
	second_sentences = [ f"{example[str(i+1)]}" for i in range(NMultChoice) ]
	flat_first = list(chain(first_sentences))
	flat_second = list(chain(second_sentences))
	tokenized_examples = tokenizer(first_sentences, second_sentences, truncation=True)
	tokDict = {k: [v[i : i + NMultChoice] for i in range(0, len(v), NMultChoice)] for k, v in tokenized_examples.items()}	        
	tokDict['label'] = example[Label_name]
	tokDict['idx'] = example[Idx_name]
	tokDict['scores'] = {f'{i}': example[f'{i+6}'] for i in range(NMultChoice)}		
	return tokDict
</code></pre>

username: anantguptadbl
cooked: <p><a class="mention" href="/u/rbelew">@rbelew</a><br>
Okay perfect, you are returning the value from <code>preprocessOne</code><br>
Since you did not have any LHS variable, I assumed that you are not returning anything from the function. You just need to consume the updated structure from the map</p>
<pre><code class="lang-auto">ds = ds.map(preprocessOne)
ds.save_to_disk(mapDir)
</code></pre>

username: rbelew
cooked: <p>ha, it was that simple!  i just had to use <code> ds2 = ds.map(preprocessOne)</code> vs. what I had assumed as in-place <code>ds.map(preprocessOne)</code>.  thanks so much <a class="mention" href="/u/anantguptadbl">@anantguptadbl</a></p>

username: anantguptadbl
cooked: <p><a class="mention" href="/u/rbelew">@rbelew</a> No problemo. Please mark it as the solution</p>

username: rbelew
cooked: <p>done!  Now I’m having some issue with DataCollators and batches, but I’ll save that for another thread. thanks again.</p>