Q:

Syntactic sugar doesn't work in metatable declaration in Lua

I'm trying to add indexing to strings like in Python.  This works:
getmetatable('').__index = function(str, i) return string.sub(str, i, i) end

str1 = 'hello'    
print(str1[1])

This doesn't:
getmetatable('').__index = function(str, i) return str:sub(i, i) end

Gives out the following error:
lua: test.lua:1: C stack overflow
stack traceback:
    test.lua:1: in function '__index'
    test.lua:1: in function '__index'
    ...
    test.lua:1: in function '__index'
    test.lua:4: in main chunk
    [C]: in ? 

Is there some sort of loop happening?  Why? 

A:

The str:method shortcut works via __index.   By re-defining __index, you break that. 
Since 5.2 or 5.3, Lua defines a metatable for strings, roughly like
debug.setmetatable( "", { __index = string } )

which permits writing ("foo"):sub( i, j ). 

Now you come along and say
getmetatable('').__index = function(str, i) return str:sub(i, i) end

so if you say ("foo")[2], that calls __index( "foo", 2 ), and inside that this causes the lookup ("foo")["sub"] (in str:sub(i, i)).   This calls __index( "foo", "sub" ), and inside that this causes the lookup ("foo")["sub"] (in str:sub(i, i)).   This calls __index( "foo", "sub" ), and…
…and the stack overflows, because you lookup ("foo")["sub"] forever. 

Re-defining __index in the way that you did means that you can no longer use that shortcut, which means you either have to spell out string.method instead of str:method everywhere in your code, library code that you use, … – or you keep it compatible. 
Character-indexing only makes sense for numbers, so you can get both by saying
getmetatable( "" ).__index = function( str, k )
    if type( k ) ~= "number" then  return string[k]  end -- lookup in string.* 
    return str:sub( k, k )
end

This will only call string.sub for a numeric index.   Method names are strings, so they will still be looked up in string.