I may be completely off-base, and would be happy to be proved wrong, but I believe you are veering close to problems that are reputedly intractable.
to simplify your simplest example even further, suppose $x $consists of two commuting matrices $a, b $.