Ultimately, this depends on the size of the bag and what makes you comfortable. The typical size of a carry-on bag is under 55-inches. If the garment bag isn’t one that is meant to compress or fold down, squeezing it into the shape of a suitcase and storing it in the overhead compartment could cause unwanted wrinkles.
Another thing you might want to consider before checking a garment bag is what is inside. If you are traveling to a destination wedding, whether you are the bride or not, you will be carrying precious cargo. For this reason, many people choose to carry their garment bags onto the plane with them and keep an eye on their belongings.