Build from source fails: undefined reference to caffe2::DeviceQuery
Labels: module: build, triaged

### 🐛 Describe the bug

Trying to build the latest version of libtorch to use it in a c++ project.

```
 cmake -D BUILD_SHARED_LIBS=ON -D CMAKE_BUILD_TYPE=Release -D USE_CUDA=ON -D USE_CUDNN=ON -D BUILD_PYTHON=OFF -D BUILD_TEST=OFF -D CMAKE_INSTALL_PREFIX=../pytorch-install -D USE_SYSTEM_NCCL=OFF -D BUILD_BINARY=ON -D USE_PRECOMPILED_HEADERS=ON -D USE_SYSTEM_EIGEN_INSTALL=ON ..

cmake --build . --target install
```

I got this error:
```
[ 76%] Building CXX object caffe2/CMakeFiles/torch_cuda.dir/__/torch/csrc/jit/passes/frozen_conv_add_relu_fusion_cuda.cpp.o
[ 76%] Building CXX object caffe2/CMakeFiles/torch_cuda.dir/__/torch/csrc/jit/tensorexpr/cuda_codegen.cpp.o
[ 76%] Building CXX object caffe2/CMakeFiles/torch_cuda.dir/__/torch/csrc/jit/runtime/register_cuda_ops.cpp.o
[ 76%] Building CUDA object caffe2/CMakeFiles/torch_cuda.dir/__/aten/src/ATen/native/cuda/Unique.cu.o
[ 76%] Linking CXX shared library ../lib/libtorch_cuda.so
[ 76%] Built target torch_cuda
[ 76%] Building CXX object caffe2/CMakeFiles/torch.dir/__/empty.cpp.o
[ 76%] Linking CXX shared library ../lib/libtorch.so
[ 76%] Built target torch
[ 76%] Built target torch_global_deps
[ 76%] Linking C executable ../../bin/mkrename_gnuabi
[ 76%] Built target mkrename_gnuabi
[ 76%] Linking C executable ../../bin/mkmasked_gnuabi
[ 76%] Built target mkmasked_gnuabi
[ 76%] Built target arraymap
[ 76%] Linking C executable ../../bin/addSuffix
[ 76%] Built target addSuffix
[ 76%] Building CXX object binaries/CMakeFiles/parallel_info.dir/parallel_info.cc.o
[ 76%] Linking CXX executable ../bin/parallel_info
[ 76%] Built target parallel_info
[ 76%] Building CXX object binaries/CMakeFiles/record_function_benchmark.dir/record_function_benchmark.cc.o
[ 76%] Linking CXX executable ../bin/record_function_benchmark
[ 76%] Built target record_function_benchmark
[ 76%] Building CXX object binaries/CMakeFiles/speed_benchmark_torch.dir/speed_benchmark_torch.cc.o
....pytorch/binaries/speed_benchmark_torch.cc: In function ‘int main(int, char**)’:
..../pytorch/binaries/speed_benchmark_torch.cc:270:33: warning: comparison of integer expressions of different signedness: ‘int’ and ‘c10::List<c10::IValue>::size_type’ {aka ‘long unsigned int’} [-Wsign-compare]
  270 |     if (FLAGS_use_bundled_input >= all_inputs.size()) {
      |         ~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~
..../pytorch/binaries/speed_benchmark_torch.cc: In instantiation of ‘c10::IValue {anonymous}::vkRunner<T>::run(T&, const std::vector<c10::IValue>&) [with T = torch::jit::Module]’:
..../pytorch/binaries/speed_benchmark_torch.cc:180:23:   required from here
..../pytorch/binaries/speed_benchmark_torch.cc:200:27: warning: comparison of integer expressions of different signedness: ‘int’ and ‘c10::List<at::Tensor>::size_type’ {aka ‘long unsigned int’} [-Wsign-compare]
  200 |           for (int i=0; i < input_as_list.size(); ++i) {
      |                         ~~^~~~~~~~~~~~~~~~~~~~~~
[ 76%] Linking CXX executable ../bin/speed_benchmark_torch
[ 76%] Built target speed_benchmark_torch
[ 76%] Building CXX object binaries/CMakeFiles/compare_models_torch.dir/compare_models_torch.cc.o
[ 76%] Linking CXX executable ../bin/compare_models_torch
[ 76%] Built target compare_models_torch
[ 76%] Building CXX object binaries/CMakeFiles/inspect_gpu.dir/inspect_gpu.cc.o
[ 76%] Linking CXX executable ../bin/inspect_gpu
/usr/bin/ld: CMakeFiles/inspect_gpu.dir/inspect_gpu.cc.o: in function `main':
inspect_gpu.cc:(.text.startup+0x4c): undefined reference to `caffe2::GlobalInit(int*, char***)'
/usr/bin/ld: inspect_gpu.cc:(.text.startup+0x1a3): undefined reference to `caffe2::DeviceQuery(int)'
/usr/bin/ld: inspect_gpu.cc:(.text.startup+0x1c5): undefined reference to `caffe2::GetCudaPeerAccessPattern(std::vector<std::vector<bool, std::allocator<bool> >, std::allocator<std::vector<bool, std::allocator<bool> > > >*)'
collect2: error: ld returned 1 exit status
make[2]: *** [binaries/CMakeFiles/inspect_gpu.dir/build.make:107: bin/inspect_gpu] Error 1
make[1]: *** [CMakeFiles/Makefile2:6382: binaries/CMakeFiles/inspect_gpu.dir/all] Error 2
make: *** [Makefile:146: all] Error 2

```

### Versions

```
PyTorch version: 1.12.0+cu116
Is debug build: False
CUDA used to build PyTorch: 11.6
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.1 LTS (fossa-b X00) (x86_64)
GCC version: (Ubuntu 10.3.0-1ubuntu1~20.04) 10.3.0
Clang version: 10.0.0-4ubuntu1 
CMake version: version 3.25.1
Libc version: glibc-2.31

Python version: 3.8.10 (default, Jun 22 2022, 20:18:18)  [GCC 9.4.0] (64-bit runtime)
Python platform: Linux-5.14.0-1054-oem-x86_64-with-glibc2.29
Is CUDA available: True
CUDA runtime version: 11.8.89
CUDA_MODULE_LOADING set to: 
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 3080 Laptop GPU
Nvidia driver version: 520.61.05
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.6.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.6.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.6.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.6.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.6.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.6.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.6.0
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.23.1
[pip3] numpy-ringbuffer==0.2.2
[pip3] pytorch-lightning==1.6.3
[pip3] pytorchcv==0.0.67
[pip3] torch==1.12.0+cu116
[pip3] torchaudio==0.12.0+cu116
[pip3] torchmetrics==0.8.2
[pip3] torchvision==0.13.0+cu116
[conda] Could not collect
```

Please ignore the pip ones and conda as they're not relevant.


cc @ezyang @seemethere @malfet