FakeTensorMode cannot handle non-fake tensor, but non-fake tensors can arise from non-interposable Tensor construction calls
Labels: triaged, oncall: pt2

### 🐛 Describe the bug

Currently, FakeTensorMode does not support non-Fake inputs in operations, except in a few special cases. These special cases typically have to do with cases where we produce tensors deep in the bowels of ATen via a non-dispatchable function, so FakeTensorMode doesn't get to interpose on the construction and we end up with a freshly generated non-fake tensor floating around.

The particular case I noticed this occurring was `at::scalar_to_tensor`, which internally bypasses the dispatcher (actually, I'm pretty sure the guards are unnecessary):

```
template <typename scalar_t>
inline void fill_inplace(Tensor& self, const Scalar& value_scalar) {
  auto value = value_scalar.to<scalar_t>();
  scalar_t* dptr = static_cast<scalar_t*>(self.data_ptr());
  *dptr = value;
}
}

namespace detail {
Tensor& scalar_fill(Tensor& self, const Scalar& value) {
  AT_DISPATCH_ALL_TYPES_AND_COMPLEX_AND4(
      kComplexHalf, kHalf, kBool, kBFloat16, self.scalar_type(), "fill_out", [&]() {
        fill_inplace<scalar_t>(self, value);
      });
  return self;
}

Tensor scalar_tensor_static(const Scalar& s, c10::optional<ScalarType> dtype_opt, c10::optional<Device> device_opt) {
  at::tracer::impl::NoTracerDispatchMode tracer_guard;
  at::AutoDispatchBelowAutograd mode;
  Tensor result = at::detail::empty_cpu(
      {}, dtype_opt, c10::nullopt, device_opt, c10::nullopt, c10::nullopt);
  scalar_fill(result, s);
  return result;
}
```

This makes it similar to `torch.tensor` in Python: a tensor constant materializes out of nowhere, and so if we want a mode to be able to interpose on it, we have to then call the result with `at::lift_fresh`. 

Now, I'm a little leery about actually inserting the dispatcher call here, because we explicitly removed the dispatch call for speed (from @ailzhang in #29915). And in fact, we only have to worry about a bare `scalar_to_tensor` call for composite ops. In https://github.com/pytorch/pytorch/pull/81609 ~~I remove `scalar_to_tensor` from autograd formulas which are interpreted as composites and was the source of the bug I was tracking down. All other usages are OK.~~ I ended up *only* calling `lift_fresh` in the derivative formulas, and not in `scalar_to_tensor` generally. Testing this feels like a beefier form of composite compliance https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/README.md#composite-compliance (cc @ezyang @zou3519 ).

The crux of the matter is this: if we don't allow FakeTensorMode to handle non-fake tensor, we must audit all composite functions for calls to non-dispatching functions that construct Tensors, and make them not call that or call lift. This includes auditing for uses of `scalar_to_tensor` in composites beyond the derivative formulas (which I have finished auditing). However, we have another option: we can make FakeTensorMode handle non-fake tensors, by implicitly adding the equivalent of a `lift_fresh` call whenever it sees an unknown tensor. However, this makes it easier for an end user of fake tensor to foot-gun themselves by passing in a non-fake tensor from the ambient context, and then attempting to mutate its metadata (which is not going to ge propagated to the original). So maybe auditing is better, esp if we can come up with a better composite compliance test.

BTW, another way tensors can be created is from saved variable. Hypothetically, a wrapped number tensor could be saved as a variable and we don't currently propagate this property.

```
diff --git a/torch/csrc/autograd/saved_variable.cpp b/torch/csrc/autograd/saved_variable.cpp
index c6ca8eda13..bfc3bd6277 100644
--- a/torch/csrc/autograd/saved_variable.cpp
+++ b/torch/csrc/autograd/saved_variable.cpp
@@ -51,6 +51,7 @@ SavedVariable::SavedVariable(
     is_leaf_ = variable.is_leaf();
     is_output_ = is_output;
     is_inplace_on_view_ = is_inplace_on_view;
+    is_wrapped_number_ = variable.unsafeGetTensorImpl()->is_wrapped_number();
 
     if (is_inplace_on_view) {
       TORCH_INTERNAL_ASSERT(!is_leaf_ && is_output);
@@ -223,6 +224,10 @@ Variable SavedVariable::unpack(std::shared_ptr<Node> saved_for) const {
     var._set_fw_grad(new_fw_grad, /* level */ 0, /* is_inplace_op */ false);
   }
 
+  if (is_wrapped_number_) {
+    var.unsafeGetTensorImpl()->set_wrapped_number(true);
+  }
+
   return var;
 }
 
diff --git a/torch/csrc/autograd/saved_variable.h b/torch/csrc/autograd/saved_variable.h
index 9136ff1a62..23f18f5b72 100644
--- a/torch/csrc/autograd/saved_variable.h
+++ b/torch/csrc/autograd/saved_variable.h
@@ -93,6 +93,7 @@ class TORCH_API SavedVariable {
   bool saved_original_ = false;
   bool is_leaf_ = false;
   bool is_output_ = false;
+  bool is_wrapped_number_ = false;
 
   // Hooks are a pair of functions pack_hook/unpack_hook that provides
   // fine-grained control over how the SavedVariable should save its data.
```

However, I could not find any case where this actually made a difference. cc @albanD 

cc @ezyang @bdhirsh @Chillee @eellison 

### Versions

master