The Sentinel feat is all based around attacks of opportunity, while Mage Slayer is checking for spellcasting and providing an attack against them. The only way these interact is that you can better disrupt their concentration when you hit an enemy with the opportunity attack.