There is no secret there but only a way to make you feel lost. Hermit's second bodyguard offers you that he reveal hermit's secret so you can defeat him easily if you pay him 450 coins. But he will disappear without telling you anything.