text: 
  
  TITLE
  [performance] module init w/ `from_pretrained` skip storage allocation
  
  COMMENTS
  0
  
  REACTIONS
  +1: 0
  -1: 0
  laugh: 0
  hooray: 0
  heart: 6
  rocket: 0
  eyes: 0
  
  BODY
  # 🚀 Feature request
  
  pt-1.9.0 added `torch.nn.utils.skip_init()` which (1) skips the module init (2) doesn't allocate any memory
  https://pytorch.org/tutorials/prototype/skip_param_init.html
  
  note: `torch.nn.utils.skip_init()` itself will be in 1.9.1, but the rest of the code should be in 1.9.0 (update: as 1.9.1 isn't planned, probably `s/1.9.1/1.10/`)
  
  We already implemented part 1 (skipping the custom init) in https://github.com/huggingface/transformers/pull/11471.
  
  We could further speed up the start up time and reduce CPU memory usage, by not allocating any storage for module init since `load_state_dict` will already have allocated `state_dict` from the pretrained weights (and some sub-modules that don't have pre-trained weights - will have to go through normal init). See https://pytorch.org/tutorials/prototype/skip_param_init.html#implementation-details
  
  another note: currently deepspeed needs to have the module storage pre-allocated for its `zero.Init` gather/scatter, but if the initial model's weights aren't allocated, then we can probably get rid of  `zero.Init` altogether https://github.com/huggingface/transformers/issues/12273