According to airline rules, if your carry-on bag is too large, the airline will require you to gate-check your bag and, often, pay a hefty fee to do so. Of course, what the rules state and what actually happens isn’t always the same. The agent might not notice, they might notice but not care, or they might notice and require you to check the bag. It can depend a bit on the airline (some are notoriously more strict than others), the agent, and how egregiously large your bag is.
If you are required to check your bag, it will be tagged and you’ll pick it up at baggage claim at your destination. Sometimes you’ll pick it up on the jetway once you’ve landed at your destination. The gate agent will tell you if that’s the case.