Kharitonov nearly qualified for the Athens Games by winning the silver medal at the 2004 Asian Amateur Boxing Championships in Puerto Princesa, Philippines. In the final he was defeated by Uzbekistan's Rustam Saidov. In the fall of 2004 he competed in the Russian Boxing Championship and placed second. Sergei could not fight in the final match due to an injury.