added a condition for the subnav variable.
if(!empty($term_slug)) {
    $subnav = wp_list_bookmarks( $args );
}

Since it was creating loops of menu, if a page is not assinged to any category.