dashed line here and the interesting thing it's hard to know if it's a coincidence or not but if you look at the in some sense the midway point between hard VQ and South vq i based on the the Ray Lu function at the hard VQ side and the sigmoid gated linear unit at the soft vq the swish is precisely halfway in between it's quite quite ok you could also pull out sigmoid hyperbolic tangent by adopting a probabilistic viewpoint of the output of a layer no longer being just a deterministic output of the input but