Yes, you can be so required, unless of course you were currently put into immediate danger while doing so. While you understandably are reluctant to attend a meeting in this building, your employer has every right to insist that you attend. Your only rights here would be if this action in some way violated the terms of a union agreemnt or employment contract. Otherwise, an employer is generally free to set the conditions of work much as it sees fit (absent some form of legally actionable discrimination, which does not appear to be the case here).