Some states have boycotted the Games on various occasions