Repository: deepchem/deepchem

Issue:
Bug in dataset.get_shape() when used after dataset.set_shard()
## 🐛 Bug

## To Reproduce

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## Expected behavior

```
import deepchem as dc
import numpy as np
X = np.random.randn(10, 3)
y = np.random.randn(10)

dataset = dc.data.DiskDataset.from_numpy(X, y)
dataset.get_shape()  # Output: ((10, 3), (10,), (10,), (10,))

for i, (X, y, w, ids) in enumerate(dataset.itershards()):
    X = X[1:]
    y = y[1:]
    w = w[1:]
    ids = ids[1:]
    dataset.set_shard(i, X, y, w, ids)

dataset.get_shape()  
# Output: ((10, 3), (10,), (10,), (10,))
# Expected output: ((9, 3), (9, ), (9, ), (9,))
```

Edit 1:

This prints correctly:
```
for i, (X, y, w, ids) in enumerate(dataset.itershards()):
    print (X.shape)  # Output: (9, 3)
```
## Environment
* DeepChem version: 2.6.0.dev

## Additional context
The fix is probably a simple one.


Discussion:
arunppsg: Hey, a very nice first contribution. Thanks for it!

I presume that `legacy_metadata` holds whether the current state of metadata is valid or not. If it is `True`, then it means that metadata has become legacy and hence invalid. Am I right here?
DingQK: > Hey, a very nice first contribution. Thanks for it!
> 
> I presume that `legacy_metadata` holds whether the current state of metadata is valid or not. If it is `True`, then it means that metadata has become legacy and hence invalid. Am I right here?

Thanks! I think that's what legacy_metadata means.
arunppsg: The CI failures were fixed in #2799 . @rbharath do we make a rebase and re-run the CI and merge or can we go ahead and directly merge?
rbharath: Let's do a rebase here to be safe. @DingQK let us know if you haven't run a rebase before and we can point you to resources
DingQK: > Let's do a rebase here to be safe. @DingQK let us know if you haven't run a rebase before and we can point you to resources

Yeah, I haven't run a rebase before and I am happy to try. I think I need resources for this action. Thank you so much!
arunppsg: I usually do it like this
1. set your current branch as the branch which you want to rebase (in your case, you have to be in `develop` branch)
2. Fetch the changes in `deepchem:master` using: `git fetch upstream`
3. Rebase your `develop` branch using: `git rebase upstream/master`
4. Push the changes to remote: `git push -f origin develop`
DingQK: Hi @arunppsg,

Thank you so much for your advice! I am not sure whether I did it correctly. Can you please check again? Thank you so much for your patience.
arunppsg: Something amiss. The reason is there are redundant commits - see, the first 3 are original commits, the last 3 are duplicate commits of the same. Make sure that you are in `develop` branch. Then, do the following:
```
git reset --hard 987afc6
git remote set-url upstream https://github.com/deepchem/deepchem.git
git fetch upstream
git rebase upstream/master
git push -f origin develop
```
Briefly, we are resetting it to commit `987afc6` and setting remote upstream url, fetching new changes from upstream, updating it to the local repo and pushing the changes. Your total commits in the PR before and after should be same (3). This [ref](https://docs.github.com/en/get-started/using-git/about-git-rebase) might help you to understand more about rebase.

Patch:
diff --git a/deepchem/data/datasets.py b/deepchem/data/datasets.py
index f80a399ca..dd5ba637f 100644
--- a/deepchem/data/datasets.py
+++ b/deepchem/data/datasets.py
@@ -2286,6 +2286,7 @@ class DiskDataset(Dataset):
     basename = "shard-%d" % shard_num
     DiskDataset.write_data_to_disk(self.data_dir, basename, X, y, w, ids)
     self._cached_shards = None
+    self.legacy_metadata = True
 
   def select(self,
              indices: Sequence[int],


Test patch:
diff --git a/deepchem/data/tests/test_setshard.py b/deepchem/data/tests/test_setshard.py
new file mode 100644
index 000000000..0fcf4b03e
--- /dev/null
+++ b/deepchem/data/tests/test_setshard.py
@@ -0,0 +1,21 @@
+import deepchem as dc
+import numpy as np
+
+
+def test_setshard_with_X_y():
+  """Test setsharding on a simple example"""
+  X = np.random.rand(10, 3)
+  y = np.random.rand(10,)
+  dataset = dc.data.DiskDataset.from_numpy(X, y)
+  X_shape, y_shape, _, _ = dataset.get_shape()
+  assert X_shape[0] == 10
+  assert y_shape[0] == 10
+  for i, (X, y, w, ids) in enumerate(dataset.itershards()):
+    X = X[1:]
+    y = y[1:]
+    w = w[1:]
+    ids = ids[1:]
+    dataset.set_shard(i, X, y, w, ids)
+  X_shape, y_shape, _, _ = dataset.get_shape()
+  assert X_shape[0] == 9
+  assert y_shape[0] == 9