[RFC] Support for Sharded Saving/Loading 
🚀 Feature
When integrating DeepSpeed/Fully Sharded, parameters are sharded across processes. This means when saving unless we consolidate onto one rank, there is the need to save on multiple processes.
In cases where the model is massive, saving to one rank is unfeasible. This happens on a per training type case (DeepSpeed/FSDP will have the option to save to one file from rank 0, or each process saves their own file).
Potential Solutions
Allow the TrainingType plugin to control the save checkpoint logic and the restore from checkpoint logic as should be done here: #6689 and in a follow-up PR
Unfortunately with the above approach when doing a load from sharded states, it's impossible to control how to load till distributed communications have been set up:
model = BoringModel()
trainer = Trainer(plugins='ddp_fully_sharded')
trainer.fit(model, dm)

# In a later session

# Fails because we can't load a sharded set of checkpoints unless distributed setup is complete
BoringModel.load_from_checkpoint('my_model')

# Works because we can set up distributed communication and load a sharded state
trainer = Trainer(plugins='ddp_fully_sharded')
trainer.predict(model, dm)
I'm unsure of other solutions outside of using the Trainer, which limits our ability to run inference/test. I think we can try to limit as much exposure the TrainingTypePlugin has, so it's only responsible for saving/final changes to the checkpoint but this still ties model saving to the plugin which may be not preferred.
cc @ananthsub @carmocca @tchaton