2048-bit keys are acceptable until 2030 as per NIST. Remember that longer keys take more time to generate and require more processing power when used for encrypting and decrypting. So, in the case of file transfer servers, if your physical server is relatively old and has limited computing resources, then 4096-bit keys may impact your server's performance.
Learn more about RSA recommended key sizes