Your mango jam might not be set if you used over-ripened mangoes. You didn't cook it enough. Just cook it longer.