Repository: microsoft/nni

Issue:
NAS Visualization doesn't work for only ENAS micro search space
**Environment**:
- NNI version: 1.7
- NNI mode (local|remote|pai): remote
- Client OS: Windows
- Server OS (for remote mode only): Ubuntu
- Python version: 3.7
- PyTorch/TensorFlow version: 1.4.0(PyTorch) 
- Is conda/virtualenv/venv used?: native env
- Is running in Docker?: Yes

NAS Visualization doesn't work for only ENAS micro search space.
It works well for ENAS macro search space

~/nni/examples/nas/enas$ python3 search.py --search-for micro --visualization
Files already downloaded and verified
Files already downloaded and verified
[08/17/2020, 11:53:18 PM] INFO (nni.nas.pytorch.trainer) Creating graph json, writing to logs/1597708397.9150221. Visualization enabled.
Traceback (most recent call last):
  File "search.py", line 60, in <module>
    trainer.enable_visualization()
  File "/home/user/.local/lib/python3.7/site-packages/nni/nas/pytorch/trainer.py", line 189, in enable_visualization
    json.dump(self.mutator.graph(sample), f)
  File "/home/user/.local/lib/python3.7/site-packages/nni/nas/pytorch/mutator.py", line 115, in graph
    graph_def, _ = build_graph(self.model, inputs, verbose=False)
  File "/home/user/.local/lib/python3.7/site-packages/nni/_graph_utils.py", line 23, in build_graph
    g = TorchProtoGraph(model, dummy_input, verbose)
  File "/home/user/.local/lib/python3.7/site-packages/nni/_graph_utils.py", line 82, in __init__
    super().__init__(model, dummy_input)
  File "/home/user/.local/lib/python3.7/site-packages/nni/_graph_utils.py", line 61, in __init__
    self._trace(model, dummy_input)
  File "/home/user/.local/lib/python3.7/site-packages/nni/_graph_utils.py", line 68, in _trace
    self.trace = torch.jit.trace(model, dummy_input)
  File "/home/user/.local/lib/python3.7/site-packages/torch/jit/__init__.py", line 882, in trace
    check_tolerance, _force_outplace, _module_class)
  File "/home/user/.local/lib/python3.7/site-packages/torch/jit/__init__.py", line 1034, in trace_module
    module._c._create_method_from_trace(method_name, func, example_inputs, var_lookup_fn, _force_outplace)
  File "/home/user/.local/lib/python3.7/site-packages/torch/nn/modules/module.py", line 530, in __call__
    result = self._slow_forward(*input, **kwargs)
  File "/home/user/.local/lib/python3.7/site-packages/torch/nn/modules/module.py", line 516, in _slow_forward
    result = self.forward(*input, **kwargs)
  File "/home/user/nni/examples/nas/enas/micro.py", line 179, in forward
    prev, cur = layer(prev, cur)
  File "/home/user/.local/lib/python3.7/site-packages/torch/nn/modules/module.py", line 530, in __call__
    result = self._slow_forward(*input, **kwargs)
  File "/home/user/.local/lib/python3.7/site-packages/torch/nn/modules/module.py", line 516, in _slow_forward
    result = self.forward(*input, **kwargs)
  File "/home/user/nni/examples/nas/enas/micro.py", line 117, in forward
    node_out, mask = self.nodes[i](prev_nodes_out)
  File "/home/user/.local/lib/python3.7/site-packages/nni/nas/pytorch/mutables.py", line 116, in __call__
    return super().__call__(*args, **kwargs)
  File "/home/user/.local/lib/python3.7/site-packages/nni/nas/pytorch/mutables.py", line 55, in __call__
    return super().__call__(*args, **kwargs)
  File "/home/user/.local/lib/python3.7/site-packages/torch/nn/modules/module.py", line 530, in __call__
    result = self._slow_forward(*input, **kwargs)
  File "/home/user/.local/lib/python3.7/site-packages/torch/nn/modules/module.py", line 516, in _slow_forward
    result = self.forward(*input, **kwargs)
  File "/home/user/nni/examples/nas/enas/micro.py", line 65, in forward
    return out_x + out_y, mask_x | mask_y
RuntimeError: cbitor is only supported for integer type tensors at /pytorch/aten/src/TH/generic/THTensorMath.cpp:66


Discussion:

Resolving patch:
diff --git a/src/sdk/pynni/nni/nas/pytorch/mutator.py b/src/sdk/pynni/nni/nas/pytorch/mutator.py
--- a/src/sdk/pynni/nni/nas/pytorch/mutator.py
+++ b/src/sdk/pynni/nni/nas/pytorch/mutator.py
@@ -162,7 +162,7 @@ def on_forward_layer_choice(self, mutable, *args, **kwargs):
         if self._connect_all:
             return self._all_connect_tensor_reduction(mutable.reduction,
                                                       [op(*args, **kwargs) for op in mutable]), \
-                torch.ones(len(mutable))
+                torch.ones(len(mutable)).bool()
 
         def _map_fn(op, args, kwargs):
             return op(*args, **kwargs)
@@ -192,7 +192,7 @@ def on_forward_input_choice(self, mutable, tensor_list):
         """
         if self._connect_all:
             return self._all_connect_tensor_reduction(mutable.reduction, tensor_list), \
-                torch.ones(mutable.n_candidates)
+                torch.ones(mutable.n_candidates).bool()
         mask = self._get_decision(mutable)
         assert len(mask) == mutable.n_candidates, \
             "Invalid mask, expected {} to be of length {}.".format(mask, mutable.n_candidates)


Test patch: