Fix a bug accessing a deferred attribute multiple times

Fix issue with deferred fields when it is already undeferred