Add test for core.nanops and fix several bugs in nanops

This pull request accomplishes two main things:

1. It adds unit tests for ```nanops```
2. It fixes several bugs identified by the unit tests

related #7352 
related #7353 
related #7354 
related #7357