Brett thought that his cute hat had finally won her over