Add new column to dataset
Opened by andysingal (closed)

### Describe the bug

```
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
[<ipython-input-9-bd197b36b6a0>](https://localhost:8080/#) in <cell line: 1>()
----> 1 dataset['train']['/workspace/data']

3 frames
[/usr/local/lib/python3.10/dist-packages/datasets/formatting/formatting.py](https://localhost:8080/#) in _check_valid_column_key(key, columns)
    518 def _check_valid_column_key(key: str, columns: List[str]) -> None:
    519     if key not in columns:
--> 520         raise KeyError(f"Column {key} not in the dataset. Current columns in the dataset: {columns}")
    521 
    522 

KeyError: "Column train not in the dataset. Current columns in the dataset: ['image', '/workspace/data']"
```

### Steps to reproduce the bug

please find the notebook for reference: https://colab.research.google.com/drive/10lZ_zLtU4itYVmIVTvIEVbjfOtCZaAZy?usp=sharing

### Expected behavior

add column to the dataset

### Environment info

colab pro

Comments:
I think it's an issue with the code.

Specifically:
```python
dataset = dataset['train'].add_column("/workspace/data", new_column)
```

Now `dataset` is the train set with a new column. 
To fix this, you can do:

```python
dataset['train'] = dataset['train'].add_column("/workspace/data", new_column)
```

> I think it's an issue with the code.
> 
> Specifically:
> 
> ```python
> dataset = dataset['train'].add_column("/workspace/data", new_column)
> ```
> 
> Now `dataset` is the train set with a new column. To fix this, you can do:
> 
> ```python
> dataset['train'] = dataset['train'].add_column("/workspace/data", new_column)
> ```

Thanks for your response, but i can not access mask images, please let me know why the problem still persists. Here is the notebook for reference: https://colab.research.google.com/drive/10lZ_zLtU4itYVmIVTvIEVbjfOtCZaAZy?usp=sharing 

I think there is a slight misunderstanding.
```python
new_column = ["mask"] * len(dataset["train"])
dataset['train'] = dataset['train'].add_column("/workspace/data", new_column)
```

adds a column with the string `mask` to your dataset.
If you're trying to load the images `"mask_{idx}.png"` in your dataset, you could try:

```
from datasets import Image

dataset['train'] = dataset['train'].map(lambda u, idx: {'mask': f"/workspace/data/mask_{idx}.png", with_indices=True).cast_column("mask", Image())
```

What this does is that it adds a column to your dataset name `mask` with the path to the mask, then it cast the column as an `Image` feature.

This [link](https://huggingface.co/docs/datasets/v2.5.1/en/image_load) explains how to load images.

Hope this helps!

> I think there is a slight misunderstanding.
> 
> ```python
> new_column = ["mask"] * len(dataset["train"])
> dataset['train'] = dataset['train'].add_column("/workspace/data", new_column)
> ```
> 
> adds a column with the string `mask` to your dataset. If you're trying to load the images `"mask_{idx}.png"` in your dataset, you could try:
> 
> ```
> from datasets import Image
> 
> dataset['train'] = dataset['train'].map(lambda u, idx: {'mask': f"/workspace/data/mask_{idx}.png", with_indices=True).cast_column("mask", Image())
> ```
> 
> What this does is that it adds a column to your dataset name `mask` with the path to the mask, then it cast the column as an `Image` feature.
> 
> This [link](https://huggingface.co/docs/datasets/v2.5.1/en/image_load) explains how to load images.
> 
> Hope this helps!

Thank you very much, this is really helpful...
i made some changes for it to work:
```
dataset['train'] = dataset['train'].map(lambda u, idx: {'mask': f"/content/data/mask_{idx}.png"}, with_indices=True).cast_column("mask", Image())
```
Thanks Again @Dref360