Q:

jQuery find next and prev element

I try to find a next or prev element of current element.  But next() and prev() function can only work in a scope, it can not reach outside.  For an example this is what I want to achieve:
<ul id="ul1">
  <li>1</li>
  <li>2</li>
  <li>3</li>
  <li>
    <ul id="ul2">
      <li>4</li>
      <li>
        <ul id="ul3">
          <li>5</li>
          <li>6</li>
        </ul>
      </li>
      <li>7</li>
      <li>8</li>
    </ul>
  </li>
  <li>9</li>
</ul>

If current element is ul1, next element is <li>1</li>, prev element is null. 
If current element is <li>1</li>, next element is <li>2</li>, prev element is ul1
If current element is <li>8</li>, next element is <li>9</li>, prev element is <li>7</li>

A:

First of all, this was a really neat problem.   Thanks for posting! 
I accomplished this with a chained jQuery statement.   Only thing is, if you run previous() on the top ul you'll get body.   Which I think technically makes sense.   However, if this isn't the desired behavior, see my update below. 
Usage: next("#ul3") returns <li>5</li>
Next:
function next(selector) {
    var $element = $(selector);

    return $element
        .children(":eq(0)")
        .add($element.next())
        .add($element.parents().filter(function() {
            return $(this).next().length > 0;
        }).next()).first();
}

Previous:
function previous(selector) {
    var $element = $(selector);

    return $element
        .prev().find("*:last")   
        .add($element.parent())    
        .add($element.prev())
        .last();     
}

Update If you want to limit the upper most node previous can be, you could do: 
function previous(selector, root) {
    var $element = $(selector);

    return $element
        .prev().find("*:last")   
        .add($element.parent())     
        .add($element.prev())
        .last().not($(root).parent());      
}

http://jsfiddle.net/andrewwhitaker/n89dz/
Another Update: Here's a jQuery plugin for convenience:
(function($) {
    $.fn.domNext = function() {
        return this
            .children(":eq(0)")
            .add(this.next())
            .add(this.parents().filter(function() {
                return $(this).next().length > 0;
            }).next()).first();        
    };

    $.fn.domPrevious = function() {
        return this
            .prev().find("*:last")   
            .add(this.parent())     
            .add(this.prev())
            .last();         
    };
})(jQuery);

Expanded example here: http://jsfiddle.net/andrewwhitaker/KzyAY/

A:

What an interesting problem!  Sounds like you're flattening the recursive structure of HTML so that it's loop-able - I can see it coming in handy. 
You can solve the problem by breaking .next() down into the following cases:

Element has children --> next() is the first child. 
Element has no children, and is not the last child --> next() is the same as jQuery's .next()
Element has no children, is last child:

Find closest parent that is not the last child --> next() is that parent's next sibling
If no such parent exists --> next() is null

Code would be (calling it flatNext to preserve original jQuery .next behaviour):
$.fn.flatNext = function () {
   if (this.children().length) return this.children().eq(0);
   if (! this.is(":last-child")) return this.next();
   if (this.closest(":not(:last-child)").length)
       return this.closest(":not(:last-child)").next();
   return $([]); // Return empty jQuery object rather than null
};

And $.fn.flatPrev should follow..

