Input:
Load module from string in python I have some code in the form of a string and would like to make a module out of it without writing to disk.
When I try using imp and a StringIO object to do this, I get:
>>> imp.load_source('my_module', '', StringIO('print "hello world"'))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: load_source() argument 3 must be file, not instance
>>> imp.load_module('my_module', StringIO('print "hello world"'), '', ('', '', 0))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: load_module arg#2 should be a file or None

How can I create the module without having an actual file?  Alternatively, how can I wrap a StringIO in a file without writing to disk?
UPDATE:
NOTE: This issue is also a problem in python3.
The code I'm trying to load is only partially trusted.  I've gone through it with ast and determined that it doesn't import anything or do anything I don't like, but I don't trust it enough to run it when I have local variables running around that could get modified, and I don't trust my own code to stay out of the way of the code I'm trying to import.
I created an empty module that only contains the following:
def load(code):
    # Delete all local variables
    globals()['code'] = code
    del locals()['code']

    # Run the code
    exec(globals()['code'])

    # Delete any global variables we've added
    del globals()['load']
    del globals()['code']

    # Copy k so we can use it
    if 'k' in locals():
        globals()['k'] = locals()['k']
        del locals()['k']

    # Copy the rest of the variables
    for k in locals().keys():
        globals()[k] = locals()[k]

Then you can import mymodule and call mymodule.load(code).  This works for me because I've ensured that the code I'm loading does not use globals.  Also, the global keyword is only a parser directive and can't refer to anything outside of the exec.
This really is way too much work to import the module without writing to disk, but if you ever want to do this, I believe it's the best way.

Output:
Here is how to import a string as a module (Python 2.x):
import sys,imp

my_code = 'a = 5'
mymodule = imp.new_module('mymodule')
exec my_code in mymodule.__dict__

In Python 3, exec is a function, so this should work:
import sys,imp

my_code = 'a = 5'
mymodule = imp.new_module('mymodule')
exec(my_code, mymodule.__dict__)

Now access the module attributes (and functions, classes etc) as:
print(mymodule.a)
>>> 5

To ignore any next attempt to import, add the module to sys:
sys.modules['mymodule'] = mymodule


Input:
Python: "TypeError: __str__ returned non-string" but still prints to output? I have this piece of code which creates a new note..WHen I try to print I get the following error even though it prints the output
Error:
C:\Python27\Basics\OOP\formytesting>python notebook.py
Memo=This is my first memo, Tag=example
Traceback (most recent call last):
  File "notebook.py", line 14, in <module>
    print(firstnote)
TypeError: __str__ returned non-string (type NoneType)

note.py
import datetime
class Note:
    def __init__(self, memo, tags):
        self.memo = memo
        self.tags = tags
        self.creation_date = datetime.date.today()

    def __str__(self):
        print('Memo={0}, Tag={1}').format(self.memo, self.tags)

if __name__ == "__main__":
    firstnote = Note('This is my first memo','example')
    print(firstnote)

Output:
Method __str__ should return string, not print.
def __str__(self):
    return 'Memo={0}, Tag={1}'.format(self.memo, self.tags)


Input:
How to find the cumulative sum of numbers in a list? time_interval=[4,6,12]

I want to sum up the numbers like [4+0, 4+6, 4+6+12] in order to get the list t=[4,10,22]. I tried:
x=0
for i in (time_interval):
    t1=time_interval[0]
    t2=time_interval[1]+t1
    t3=time_interval[2]+t2
    print(t1,t2,t3)

4 10 22
4 10 22
4 10 22

Output:
If you're doing much numerical work with arrays like this, I'd suggest numpy, which comes with a cumulative sum function cumsum:
import numpy as np

a = [4,6,12]

np.cumsum(a)
#array([4, 10, 22])

Numpy is often faster than pure python for this kind of thing, see in comparison to @Ashwini's accumu:
In [136]: timeit list(accumu(range(1000)))
10000 loops, best of 3: 161 us per loop

In [137]: timeit list(accumu(xrange(1000)))
10000 loops, best of 3: 147 us per loop

In [138]: timeit np.cumsum(np.arange(1000))
100000 loops, best of 3: 10.1 us per loop

But of course if it's the only place you'll use numpy, it might not be worth having a dependence on it.


Input:
Python: Pass or Sleep for long running processes? I am writing an queue processing application which uses threads for waiting on and responding to queue messages to be delivered to the app.  For the main part of the application, it just needs to stay active.  For a code example like:

while True:
  pass

or

while True:
  time.sleep(1)

Which one will have the least impact on a system?  What is the preferred way to do nothing, but keep a python app running?

Output:
I would imagine time.sleep() will have less overhead on the system. Using pass will cause the loop to immediately re-evaluate and peg the CPU, whereas using time.sleep will allow the execution to be temporarily suspended.
EDIT: just to prove the point, if you launch the python interpreter and run this:
>>> while True:
...     pass
...

You can watch Python start eating up 90-100% CPU instantly, versus:
>>> import time
>>> while True:
...     time.sleep(1)
...

Which barely even registers on the Activity Monitor (using OS X here but it should be the same for every platform).


Input: Determining what version of Flask is installed What's the easiest way to determine which version of Flask is installed?

Output:
As of flask 0.7 (June 28th, 2011), a __version__ attribute can be found on the flask module.
>> import flask
>> flask.__version__

Keep in mind that because prior to flask 0.7 there was no __version__ attribute, the preceding code will result in an attribute error on those older versions.
For versions older than flask 0.7, you might be able to determine it using pkg_resources as shown below:

>>> import pkg_resources
>>> pkg_resources.get_distribution('flask').version
'0.6.1'

This won't work 100% though. It depends on the user having the pkg_resources library installed (it might come by default with a Linux distribution's python installation, but since it's not part of the standard library you can't be positive), and also that the user installed flask in a way that pkg_resources can find it (for example, just copying the full flask source code into your directory puts it out of the range of pkg_resources).


Input:
Django returns 403 error when sending a POST request when I'm using following Python code to send a POST request to my Django website I'm getting 403: Forbidden error.
url = 'http://www.sub.domain.com/'
values = { 'var': 'test' }

try:
    data = urllib.urlencode(values, doseq=True)
    req = urllib2.Request(url, data)
    response = urllib2.urlopen(req)
    the_page = response.read()
except:
    the_page = sys.exc_info()
    raise

When I'm opening any other website it works properly.
domain.com is Django website too, and it works properly too.
I think, that's Django config problem, can anyone tell me what should I do to provide access to my script?

Output:
Look here https://docs.djangoproject.com/en/dev/ref/csrf/#how-to-use-it.
Try marking your view with @csrf_exempt. That way, Django's CSRF middleware will ignore CSRF protection. You'll also need to use from django.views.decorators.csrf import csrf_exempt. See: https://docs.djangoproject.com/en/dev/ref/csrf/#utilities
Please be advised that by disabling CSRF protection on your view, you are opening a gate for CSRF attacks.
If security is vital to you then consider using @csrf_exempt followed by @requires_csrf_token (see: https://docs.djangoproject.com/en/dev/ref/csrf/#unprotected-view-needs-the-csrf-token). Then, in your script pass this token and that's it.


Input:
interprocess communication in python What is a clean and elegant way to do interprocess communication between two different python processes?  I currently use named pipes in the OS, but it feels a bit hacky.  I rewrote my stuff with dbus services, which worked, but it seems when running the code remotely through an SSH session it now tries to initialise X11 which seems completely unnecessary for the things I want to do (they aren't GUI related).  So maybe dbus is a bit too heavyweight.  I was about to redesign again using sockets, but it seems quite low-level so I thought there might be a higher level module I could import and use which I simply don't know the name of, and I thought I should ask on SO first..
My requirement is to be able to run python foo.py and have that process just doing it's thing there, like a daemon, and be able to send messages to it with python foo.py --bar.  The latter call should just sends a message to the existing process and terminates, possibly with a return code 0 for success or other for failure (so some two-way communication will be required).

Output:
The multiprocessing library provides listeners and clients that wrap sockets and allow you to pass arbitrary python objects.
Your server could listen to receive python objects:
from multiprocessing.connection import Listener

address = ('localhost', 6000)     # family is deduced to be 'AF_INET'
listener = Listener(address, authkey='secret password')
conn = listener.accept()
print 'connection accepted from', listener.last_accepted
while True:
    msg = conn.recv()
    # do something with msg
    if msg == 'close':
        conn.close()
        break
listener.close()

Your client could send commands as objects:
from multiprocessing.connection import Client

address = ('localhost', 6000)
conn = Client(address, authkey='secret password')
conn.send('close')
# can also send arbitrary objects:
# conn.send(['a', 2.5, None, int, sum])
conn.close()


Input:
Add number to set What am I doing wrong here?
a = set().add(1)
print a # Prints `None`

I'm trying to add the number 1 to the empty set.

Output:
It is a convention in Python that methods that mutate sequences return None.
Consider:
>>> a_list = [3, 2, 1]
>>> print a_list.sort()
None
>>> a_list
[1, 2, 3]

>>> a_dict = {}
>>> print a_dict.__setitem__('a', 1)
None
>>> a_dict
{'a': 1}

>>> a_set = set()
>>> print a_set.add(1)
None
>>> a_set
set([1])

Some may consider this convention "a horrible misdesign in Python", but the Design and History FAQ gives the reasoning behind this design decision (with respect to lists):

Why doesnât list.sort() return the sorted list?
In situations where performance matters, making a copy of the list
  just to sort it would be wasteful. Therefore, list.sort() sorts the
  list in place. In order to remind you of that fact, it does not return
  the sorted list. This way, you wonât be fooled into accidentally
  overwriting a list when you need a sorted copy but also need to keep
  the unsorted version around.
In Python 2.4 a new built-in function â sorted() â has been added.
  This function creates a new list from a provided iterable, sorts it
  and returns it.

Your particular problems with this feature come from a misunderstanding of good ways to create a set rather than a language misdesign.  As Lattyware points out, in Python versions 2.7 and later you can use a set literal a = {1} or do a = set([1]) as per Sven Marnach's answer.
Parenthetically, I like Ruby's convention of placing an exclamation point after methods that mutate objects, but I find Python's approach acceptable.


Input:
Why won't Django use IPython? (myvenv)me:src orokusaki$ python manage.py shell -i ipython
Python 2.7.2 (default, Jun 16 2012, 12:38:40)
[GCC 4.2.1 Compatible Apple Clang 4.0 (tags/Apple/clang-418.0.60)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> while True:
...     pass  # :(
...

I have IPython installed globally, and it works while this myvenv virtualenv is activated just fine. However, if I start the Django shell (with or without -i), it won't use IPython. I've never had this problem before.
Note: django-admin.py vs manage.py won't make a difference.

Output: Try installing it into virtualenv! :-)


Input: Can you change a field label in the Django Admin application? As the title suggests. I want to be able to change the label of a single field in the admin application. I'm aware of the Form.field attribute, but how do I get my Model or ModelAdmin to pass along that information?

Output: the verbose name of the field is the (optional) first parameter at field construction.


Input:
How to fetch a non-ascii url with Python urlopen? I need to fetch data from a URL with non-ascii characters but urllib2.urlopen refuses to open the resource and raises:
UnicodeEncodeError: 'ascii' codec can't encode character u'\u0131' in position 26: ordinal not in range(128)

I know the URL is not standards compliant but I have no chance to change it.
What is the way to access a resource pointed by a URL containing non-ascii characters using Python?
edit: In other words, can / how urlopen open a URL like:
http://example.org/ÃÃ¶Ã±-ÃÅÃÄ°Ä°/

Output:
Strictly speaking URIs can't contain non-ASCII characters; what you have there is an IRI.
To convert an IRI to a plain ASCII URI:

non-ASCII characters in the hostname part of the address have to be encoded using the Punycode-based IDNA algorithm;
non-ASCII characters in the path, and most of the other parts of the address have to be encoded using UTF-8 and %-encoding, as per Ignacio's answer.

So:
import re, urlparse

def urlEncodeNonAscii(b):
    return re.sub('[\x80-\xFF]', lambda c: '%%%02x' % ord(c.group(0)), b)

def iriToUri(iri):
    parts= urlparse.urlparse(iri)
    return urlparse.urlunparse(
        part.encode('idna') if parti==1 else urlEncodeNonAscii(part.encode('utf-8'))
        for parti, part in enumerate(parts)
    )

>>> iriToUri(u'http://www.a\u0131b.com/a\u0131b')
'http://www.xn--ab-hpa.com/a%c4%b1b'

(Technically this still isn't quite good enough in the general case because urlparse doesn't split away any user:pass@ prefix or :port suffix on the hostname. Only the hostname part should be IDNA encoded. It's easier to encode using normal urllib.quote and .encode('idna') at the time you're constructing a URL than to have to pull an IRI apart.)


Input: List database tables with SQLAlchemy I want to implement a function that gives information about all the tables (and their column names) that are present in a database (not only those created with SQLAlchemy). While reading the documentation it seems to me that this is done via reflection but I didn't manage to get something working. Any suggestions or examples on how to do this?

Output:
start with an engine:
from sqlalchemy import create_engine
engine = create_engine("postgresql://u:p@host/database")

quick path to all table /column names, use an inspector:
from sqlalchemy import inspect
inspector = inspect(engine)

for table_name in inspector.get_table_names():
   for column in inspector.get_columns(table_name):
       print("Column: %s" % column['name'])

docs: http://docs.sqlalchemy.org/en/rel_0_9/core/reflection.html?highlight=inspector#fine-grained-reflection-with-inspector
alternatively, use MetaData / Tables:
from sqlalchemy import MetaData
m = MetaData()
m.reflect(engine)
for table in m.tables.values():
    print(table.name)
    for column in table.c:
        print(column.name)

docs: http://docs.sqlalchemy.org/en/rel_0_9/core/reflection.html#reflecting-all-tables-at-once


Input:
Optimize the performance of dictionary membership for a list of Keys I am trying to write a code which should return true if any element of list is present in a dictionary. Performance of this piece is really important. I know I can just loop over list and break if I find the first search hit. Is there any faster or more Pythonic way for this than given below?
for x in someList:
     if x in someDict:
           return True
return False

EDIT: I am using Python 2.7. My first preference would be a faster method.

Output:
Use of builtin any can have some performance edge over two loops
any(x in someDict for x in someList)

but you might need to measure your mileage. If your list and dict remains pretty static and you have to perform the comparison multiple times, you may consider using set
someSet = set(someList)
someDict.viewkeys() & someSet

Note Python 3.X, by default returns views rather than a sequence, so it would be straight forward when using Python 3.X
someSet = set(someList)
someDict.keys() & someSet

In both the above cases you can wrap the result with a bool to get a boolean result
bool(someDict.keys() & set(someSet ))

Heretic Note
My curiosity got the better of me and I timed all the proposed solutions. It seems that your original solution is better performance wise. Here is the result
Sample Randomly generated Input
def test_data_gen():
    from random import sample
    for i in range(1,5):
        n = 10**i
        population = set(range(1,100000))
        some_list = sample(list(population),n)
        population.difference_update(some_list)
        some_dict = dict(zip(sample(population,n),
                             sample(range(1,100000),n)))
        yield "Population Size of {}".format(n), (some_list, some_dict), {}

The Test Engine
I rewrote the Test Part of the answer as it was messy and the answer was receiving quite a decent attention. I created a timeit compare python module and moved it onto github
The Test Result
Timeit repeated for 10 times
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
======================================
Test Run for Population Size of 10
======================================
|Rank  |FunctionName         |Result    |Description
+------+---------------------+----------+-----------------------------------------------
|     1|foo_nested           |0.000011  |Original OPs Code
+------+---------------------+----------+-----------------------------------------------
|     2|foo_ifilter_any      |0.000014  |any(ifilter(some_dict.__contains__, some_list))
+------+---------------------+----------+-----------------------------------------------
|     3|foo_ifilter_not_not  |0.000015  |not not next(ifilter(some_dict.__contains__...
+------+---------------------+----------+-----------------------------------------------
|     4|foo_imap_any         |0.000018  |any(imap(some_dict.__contains__, some_list))
+------+---------------------+----------+-----------------------------------------------
|     5|foo_any              |0.000019  |any(x in some_dict for x in some_list)
+------+---------------------+----------+-----------------------------------------------
|     6|foo_ifilter_next     |0.000022  |bool(next(ifilter(some_dict.__contains__...
+------+---------------------+----------+-----------------------------------------------
|     7|foo_set_ashwin       |0.000024  |not set(some_dct).isdisjoint(some_lst)
+------+---------------------+----------+-----------------------------------------------
|     8|foo_set              |0.000047  |some_dict.viewkeys() & set(some_list )
======================================
Test Run for Population Size of 100
======================================
|Rank  |FunctionName         |Result    |Description
+------+---------------------+----------+-----------------------------------------------
|     1|foo_ifilter_any      |0.000071  |any(ifilter(some_dict.__contains__, some_list))
+------+---------------------+----------+-----------------------------------------------
|     2|foo_nested           |0.000072  |Original OPs Code
+------+---------------------+----------+-----------------------------------------------
|     3|foo_ifilter_not_not  |0.000073  |not not next(ifilter(some_dict.__contains__...
+------+---------------------+----------+-----------------------------------------------
|     4|foo_ifilter_next     |0.000076  |bool(next(ifilter(some_dict.__contains__...
+------+---------------------+----------+-----------------------------------------------
|     5|foo_imap_any         |0.000082  |any(imap(some_dict.__contains__, some_list))
+------+---------------------+----------+-----------------------------------------------
|     6|foo_any              |0.000092  |any(x in some_dict for x in some_list)
+------+---------------------+----------+-----------------------------------------------
|     7|foo_set_ashwin       |0.000170  |not set(some_dct).isdisjoint(some_lst)
+------+---------------------+----------+-----------------------------------------------
|     8|foo_set              |0.000638  |some_dict.viewkeys() & set(some_list )
======================================
Test Run for Population Size of 1000
======================================
|Rank  |FunctionName         |Result    |Description
+------+---------------------+----------+-----------------------------------------------
|     1|foo_ifilter_not_not  |0.000746  |not not next(ifilter(some_dict.__contains__...
+------+---------------------+----------+-----------------------------------------------
|     2|foo_ifilter_any      |0.000746  |any(ifilter(some_dict.__contains__, some_list))
+------+---------------------+----------+-----------------------------------------------
|     3|foo_ifilter_next     |0.000752  |bool(next(ifilter(some_dict.__contains__...
+------+---------------------+----------+-----------------------------------------------
|     4|foo_nested           |0.000771  |Original OPs Code
+------+---------------------+----------+-----------------------------------------------
|     5|foo_set_ashwin       |0.000838  |not set(some_dct).isdisjoint(some_lst)
+------+---------------------+----------+-----------------------------------------------
|     6|foo_imap_any         |0.000842  |any(imap(some_dict.__contains__, some_list))
+------+---------------------+----------+-----------------------------------------------
|     7|foo_any              |0.000933  |any(x in some_dict for x in some_list)
+------+---------------------+----------+-----------------------------------------------
|     8|foo_set              |0.001702  |some_dict.viewkeys() & set(some_list )
======================================
Test Run for Population Size of 10000
======================================
|Rank  |FunctionName         |Result    |Description
+------+---------------------+----------+-----------------------------------------------
|     1|foo_nested           |0.007195  |Original OPs Code
+------+---------------------+----------+-----------------------------------------------
|     2|foo_ifilter_next     |0.007410  |bool(next(ifilter(some_dict.__contains__...
+------+---------------------+----------+-----------------------------------------------
|     3|foo_ifilter_any      |0.007491  |any(ifilter(some_dict.__contains__, some_list))
+------+---------------------+----------+-----------------------------------------------
|     4|foo_ifilter_not_not  |0.007671  |not not next(ifilter(some_dict.__contains__...
+------+---------------------+----------+-----------------------------------------------
|     5|foo_set_ashwin       |0.008385  |not set(some_dct).isdisjoint(some_lst)
+------+---------------------+----------+-----------------------------------------------
|     6|foo_imap_any         |0.011327  |any(imap(some_dict.__contains__, some_list))
+------+---------------------+----------+-----------------------------------------------
|     7|foo_any              |0.011533  |any(x in some_dict for x in some_list)
+------+---------------------+----------+-----------------------------------------------
|     8|foo_set              |0.018313  |some_dict.viewkeys() & set(some_list )
Timeit repeated for 100 times
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
======================================
Test Run for Population Size of 10
======================================
|Rank  |FunctionName         |Result    |Description
+------+---------------------+----------+-----------------------------------------------
|     1|foo_nested           |0.000098  |Original OPs Code
+------+---------------------+----------+-----------------------------------------------
|     2|foo_ifilter_any      |0.000124  |any(ifilter(some_dict.__contains__, some_list))
+------+---------------------+----------+-----------------------------------------------
|     3|foo_ifilter_not_not  |0.000131  |not not next(ifilter(some_dict.__contains__...
+------+---------------------+----------+-----------------------------------------------
|     4|foo_imap_any         |0.000142  |any(imap(some_dict.__contains__, some_list))
+------+---------------------+----------+-----------------------------------------------
|     5|foo_ifilter_next     |0.000151  |bool(next(ifilter(some_dict.__contains__...
+------+---------------------+----------+-----------------------------------------------
|     6|foo_any              |0.000158  |any(x in some_dict for x in some_list)
+------+---------------------+----------+-----------------------------------------------
|     7|foo_set_ashwin       |0.000186  |not set(some_dct).isdisjoint(some_lst)
+------+---------------------+----------+-----------------------------------------------
|     8|foo_set              |0.000496  |some_dict.viewkeys() & set(some_list )
======================================
Test Run for Population Size of 100
======================================
|Rank  |FunctionName         |Result    |Description
+------+---------------------+----------+-----------------------------------------------
|     1|foo_ifilter_any      |0.000661  |any(ifilter(some_dict.__contains__, some_list))
+------+---------------------+----------+-----------------------------------------------
|     2|foo_ifilter_not_not  |0.000677  |not not next(ifilter(some_dict.__contains__...
+------+---------------------+----------+-----------------------------------------------
|     3|foo_nested           |0.000683  |Original OPs Code
+------+---------------------+----------+-----------------------------------------------
|     4|foo_ifilter_next     |0.000684  |bool(next(ifilter(some_dict.__contains__...
+------+---------------------+----------+-----------------------------------------------
|     5|foo_imap_any         |0.000762  |any(imap(some_dict.__contains__, some_list))
+------+---------------------+----------+-----------------------------------------------
|     6|foo_any              |0.000854  |any(x in some_dict for x in some_list)
+------+---------------------+----------+-----------------------------------------------
|     7|foo_set_ashwin       |0.001291  |not set(some_dct).isdisjoint(some_lst)
+------+---------------------+----------+-----------------------------------------------
|     8|foo_set              |0.005018  |some_dict.viewkeys() & set(some_list )
======================================
Test Run for Population Size of 1000
======================================
|Rank  |FunctionName         |Result    |Description
+------+---------------------+----------+-----------------------------------------------
|     1|foo_ifilter_any      |0.007585  |any(ifilter(some_dict.__contains__, some_list))
+------+---------------------+----------+-----------------------------------------------
|     2|foo_nested           |0.007713  |Original OPs Code
+------+---------------------+----------+-----------------------------------------------
|     3|foo_set_ashwin       |0.008256  |not set(some_dct).isdisjoint(some_lst)
+------+---------------------+----------+-----------------------------------------------
|     4|foo_ifilter_not_not  |0.008526  |not not next(ifilter(some_dict.__contains__...
+------+---------------------+----------+-----------------------------------------------
|     5|foo_any              |0.009422  |any(x in some_dict for x in some_list)
+------+---------------------+----------+-----------------------------------------------
|     6|foo_ifilter_next     |0.010259  |bool(next(ifilter(some_dict.__contains__...
+------+---------------------+----------+-----------------------------------------------
|     7|foo_imap_any         |0.011414  |any(imap(some_dict.__contains__, some_list))
+------+---------------------+----------+-----------------------------------------------
|     8|foo_set              |0.019862  |some_dict.viewkeys() & set(some_list )
======================================
Test Run for Population Size of 10000
======================================
|Rank  |FunctionName         |Result    |Description
+------+---------------------+----------+-----------------------------------------------
|     1|foo_imap_any         |0.082221  |any(imap(some_dict.__contains__, some_list))
+------+---------------------+----------+-----------------------------------------------
|     2|foo_ifilter_any      |0.083573  |any(ifilter(some_dict.__contains__, some_list))
+------+---------------------+----------+-----------------------------------------------
|     3|foo_nested           |0.095736  |Original OPs Code
+------+---------------------+----------+-----------------------------------------------
|     4|foo_set_ashwin       |0.103427  |not set(some_dct).isdisjoint(some_lst)
+------+---------------------+----------+-----------------------------------------------
|     5|foo_ifilter_next     |0.104589  |bool(next(ifilter(some_dict.__contains__...
+------+---------------------+----------+-----------------------------------------------
|     6|foo_ifilter_not_not  |0.117974  |not not next(ifilter(some_dict.__contains__...
+------+---------------------+----------+-----------------------------------------------
|     7|foo_any              |0.127739  |any(x in some_dict for x in some_list)
+------+---------------------+----------+-----------------------------------------------
|     8|foo_set              |0.208228  |some_dict.viewkeys() & set(some_list )
Timeit repeated for 1000 times
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
======================================
Test Run for Population Size of 10
======================================
|Rank  |FunctionName         |Result    |Description
+------+---------------------+----------+-----------------------------------------------
|     1|foo_nested           |0.000953  |Original OPs Code
+------+---------------------+----------+-----------------------------------------------
|     2|foo_ifilter_any      |0.001134  |any(ifilter(some_dict.__contains__, some_list))
+------+---------------------+----------+-----------------------------------------------
|     3|foo_ifilter_not_not  |0.001213  |not not next(ifilter(some_dict.__contains__...
+------+---------------------+----------+-----------------------------------------------
|     4|foo_ifilter_next     |0.001340  |bool(next(ifilter(some_dict.__contains__...
+------+---------------------+----------+-----------------------------------------------
|     5|foo_imap_any         |0.001407  |any(imap(some_dict.__contains__, some_list))
+------+---------------------+----------+-----------------------------------------------
|     6|foo_any              |0.001535  |any(x in some_dict for x in some_list)
+------+---------------------+----------+-----------------------------------------------
|     7|foo_set_ashwin       |0.002252  |not set(some_dct).isdisjoint(some_lst)
+------+---------------------+----------+-----------------------------------------------
|     8|foo_set              |0.004701  |some_dict.viewkeys() & set(some_list )
======================================
Test Run for Population Size of 100
======================================
|Rank  |FunctionName         |Result    |Description
+------+---------------------+----------+-----------------------------------------------
|     1|foo_ifilter_any      |0.006209  |any(ifilter(some_dict.__contains__, some_list))
+------+---------------------+----------+-----------------------------------------------
|     2|foo_ifilter_next     |0.006411  |bool(next(ifilter(some_dict.__contains__...
+------+---------------------+----------+-----------------------------------------------
|     3|foo_ifilter_not_not  |0.006657  |not not next(ifilter(some_dict.__contains__...
+------+---------------------+----------+-----------------------------------------------
|     4|foo_nested           |0.006727  |Original OPs Code
+------+---------------------+----------+-----------------------------------------------
|     5|foo_imap_any         |0.007562  |any(imap(some_dict.__contains__, some_list))
+------+---------------------+----------+-----------------------------------------------
|     6|foo_any              |0.008262  |any(x in some_dict for x in some_list)
+------+---------------------+----------+-----------------------------------------------
|     7|foo_set_ashwin       |0.012260  |not set(some_dct).isdisjoint(some_lst)
+------+---------------------+----------+-----------------------------------------------
|     8|foo_set              |0.046773  |some_dict.viewkeys() & set(some_list )
======================================
Test Run for Population Size of 1000
======================================
|Rank  |FunctionName         |Result    |Description
+------+---------------------+----------+-----------------------------------------------
|     1|foo_ifilter_not_not  |0.071888  |not not next(ifilter(some_dict.__contains__...
+------+---------------------+----------+-----------------------------------------------
|     2|foo_ifilter_next     |0.072150  |bool(next(ifilter(some_dict.__contains__...
+------+---------------------+----------+-----------------------------------------------
|     3|foo_nested           |0.073382  |Original OPs Code
+------+---------------------+----------+-----------------------------------------------
|     4|foo_ifilter_any      |0.075698  |any(ifilter(some_dict.__contains__, some_list))
+------+---------------------+----------+-----------------------------------------------
|     5|foo_set_ashwin       |0.077367  |not set(some_dct).isdisjoint(some_lst)
+------+---------------------+----------+-----------------------------------------------
|     6|foo_imap_any         |0.090623  |any(imap(some_dict.__contains__, some_list))
+------+---------------------+----------+-----------------------------------------------
|     7|foo_any              |0.093301  |any(x in some_dict for x in some_list)
+------+---------------------+----------+-----------------------------------------------
|     8|foo_set              |0.177051  |some_dict.viewkeys() & set(some_list )
======================================
Test Run for Population Size of 10000
======================================
|Rank  |FunctionName         |Result    |Description
+------+---------------------+----------+-----------------------------------------------
|     1|foo_nested           |0.701317  |Original OPs Code
+------+---------------------+----------+-----------------------------------------------
|     2|foo_ifilter_next     |0.706156  |bool(next(ifilter(some_dict.__contains__...
+------+---------------------+----------+-----------------------------------------------
|     3|foo_ifilter_any      |0.723368  |any(ifilter(some_dict.__contains__, some_list))
+------+---------------------+----------+-----------------------------------------------
|     4|foo_ifilter_not_not  |0.746650  |not not next(ifilter(some_dict.__contains__...
+------+---------------------+----------+-----------------------------------------------
|     5|foo_set_ashwin       |0.776704  |not set(some_dct).isdisjoint(some_lst)
+------+---------------------+----------+-----------------------------------------------
|     6|foo_imap_any         |0.832117  |any(imap(some_dict.__contains__, some_list))
+------+---------------------+----------+-----------------------------------------------
|     7|foo_any              |0.881777  |any(x in some_dict for x in some_list)
+------+---------------------+----------+-----------------------------------------------
|     8|foo_set              |1.665962  |some_dict.viewkeys() & set(some_list )
Timeit repeated for 10000 times
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
======================================
Test Run for Population Size of 10
======================================
|Rank  |FunctionName         |Result    |Description
+------+---------------------+----------+-----------------------------------------------
|     1|foo_nested           |0.010581  |Original OPs Code
+------+---------------------+----------+-----------------------------------------------
|     2|foo_ifilter_any      |0.013512  |any(ifilter(some_dict.__contains__, some_list))
+------+---------------------+----------+-----------------------------------------------
|     3|foo_imap_any         |0.015321  |any(imap(some_dict.__contains__, some_list))
+------+---------------------+----------+-----------------------------------------------
|     4|foo_ifilter_not_not  |0.017680  |not not next(ifilter(some_dict.__contains__...
+------+---------------------+----------+-----------------------------------------------
|     5|foo_ifilter_next     |0.019334  |bool(next(ifilter(some_dict.__contains__...
+------+---------------------+----------+-----------------------------------------------
|     6|foo_any              |0.026274  |any(x in some_dict for x in some_list)
+------+---------------------+----------+-----------------------------------------------
|     7|foo_set_ashwin       |0.030881  |not set(some_dct).isdisjoint(some_lst)
+------+---------------------+----------+-----------------------------------------------
|     8|foo_set              |0.053605  |some_dict.viewkeys() & set(some_list )
======================================
Test Run for Population Size of 100
======================================
|Rank  |FunctionName         |Result    |Description
+------+---------------------+----------+-----------------------------------------------
|     1|foo_nested           |0.070194  |Original OPs Code
+------+---------------------+----------+-----------------------------------------------
|     2|foo_ifilter_not_not  |0.078524  |not not next(ifilter(some_dict.__contains__...
+------+---------------------+----------+-----------------------------------------------
|     3|foo_ifilter_any      |0.079499  |any(ifilter(some_dict.__contains__, some_list))
+------+---------------------+----------+-----------------------------------------------
|     4|foo_imap_any         |0.087349  |any(imap(some_dict.__contains__, some_list))
+------+---------------------+----------+-----------------------------------------------
|     5|foo_ifilter_next     |0.093970  |bool(next(ifilter(some_dict.__contains__...
+------+---------------------+----------+-----------------------------------------------
|     6|foo_any              |0.097948  |any(x in some_dict for x in some_list)
+------+---------------------+----------+-----------------------------------------------
|     7|foo_set_ashwin       |0.130725  |not set(some_dct).isdisjoint(some_lst)
+------+---------------------+----------+-----------------------------------------------
|     8|foo_set              |0.480841  |some_dict.viewkeys() & set(some_list )
======================================
Test Run for Population Size of 1000
======================================
|Rank  |FunctionName         |Result    |Description
+------+---------------------+----------+-----------------------------------------------
|     1|foo_ifilter_any      |0.754491  |any(ifilter(some_dict.__contains__, some_list))
+------+---------------------+----------+-----------------------------------------------
|     2|foo_ifilter_not_not  |0.756253  |not not next(ifilter(some_dict.__contains__...
+------+---------------------+----------+-----------------------------------------------
|     3|foo_ifilter_next     |0.771382  |bool(next(ifilter(some_dict.__contains__...
+------+---------------------+----------+-----------------------------------------------
|     4|foo_nested           |0.787152  |Original OPs Code
+------+---------------------+----------+-----------------------------------------------
|     5|foo_set_ashwin       |0.818520  |not set(some_dct).isdisjoint(some_lst)
+------+---------------------+----------+-----------------------------------------------
|     6|foo_imap_any         |0.902947  |any(imap(some_dict.__contains__, some_list))
+------+---------------------+----------+-----------------------------------------------
|     7|foo_any              |1.001810  |any(x in some_dict for x in some_list)
+------+---------------------+----------+-----------------------------------------------
|     8|foo_set              |2.012781  |some_dict.viewkeys() & set(some_list )
=======================================
Test Run for Population Size of 10000
=======================================
|Rank  |FunctionName         |Result     |Description
+------+---------------------+-----------+-----------------------------------------------
|     1|foo_imap_any         |10.071469  |any(imap(some_dict.__contains__, some_list))
+------+---------------------+-----------+-----------------------------------------------
|     2|foo_any              |11.127034  |any(x in some_dict for x in some_list)
+------+---------------------+-----------+-----------------------------------------------
|     3|foo_set              |18.881414  |some_dict.viewkeys() & set(some_list )
+------+---------------------+-----------+-----------------------------------------------
|     4|foo_nested           |8.731133   |Original OPs Code
+------+---------------------+-----------+-----------------------------------------------
|     5|foo_ifilter_not_not  |9.019190   |not not next(ifilter(some_dict.__contains__...
+------+---------------------+-----------+-----------------------------------------------
|     6|foo_ifilter_next     |9.189966   |bool(next(ifilter(some_dict.__contains__...
+------+---------------------+-----------+-----------------------------------------------
|     7|foo_set_ashwin       |9.363886   |not set(some_dct).isdisjoint(some_lst)
+------+---------------------+-----------+-----------------------------------------------
|     8|foo_ifilter_any      |9.442759   |any(ifilter(some_dict.__contains__, some_list))

And a Graphical Comparison from the above referred module

Conclusion
Premature optimization is evil. It is evident that none of the solutions have optimal performance when the test domain varies. Depending on population size and frequency of iteration, performance of solutions varies considerably. The result again speaks out about the fact that in Python, one should ensure that the code should be readable rather than ensuring that the code is either nifty or optimized for performance for certain cases, but then it may not be scalable.
Note There were some doubts on why not using ifilter performs better than the rest

"In Abhit's answer, he timed the different approaches and found that
  ifilter/next was not the fastest; any idea why this would be the case?
  "

It is a known fact that in python, there is an overhead when calling C functions, and if the population size is low but the frequency of iteration is high, the accumulation of C function call overhead would slowly show up. As can be seen in the graphs, where population size is low but iteration is high, using ifilter, performance deviates considerably.


Input:
Nonalphanumeric list order from os.listdir() in Python I often use python to process directories of data. Recently, I have noticed that the default order of the lists has changed to something almost nonsensical. For example, if I am in a current directory containing the following subdirectories: run01, run02, ... run19, run20, and then I generate a list from the following command:
dir = os.listdir(os.getcwd())

then I usually get a list in this order:
dir = ['run01', 'run18', 'run14', 'run13', 'run12', 'run11', 'run08', ... ]

and so on. The order used to be alphanumeric. But this new order has remained with me for a while now.
What is determining the (displayed) order of these lists? Is there some sort of rule here? Is it the result of some setting I obliviously changed?
EDIT: Thank you for the suggestions everyone, does anyone know where/how the filesystem determines this order? I've determined that this only occurs on a MacFUSE mounted drive.

Output:
You can use the builtin sorted function to sort the strings however you want.  Based on what you describe,
sorted(os.listdir(whatever_directory))

Alternatively, you can use the .sort method of a list:
lst = os.listdir(whatever_directory)
lst.sort()

I think should do the trick.
Note that the order that os.listdir gets the filenames is probably completely dependent on your filesystem.


Input: Automatically import models on Django shell launch I'm tired of typing from account_import.models import ImportFile every time I open my Django shell. Is there a way to automatically run this command whenever I fire up my shell?

Output:
install django-extensions, one of the commands it features (shell_plus) is providing the context for your models.
https://github.com/django-extensions/django-extensions
So, instead of ./manage.py shell you can use ./manage.py shell_plus so that everything is imported.


Input:
Slicing a list in Python without generating a copy I have the following problem.

Given a list of integers L, I need to generate all of the sublists L[k:] for k in [0, len(L) - 1], without generating copies.

How do I accomplish this in Python? With a buffer object somehow?

Output:
Slicing lists does not generate copies of the objects in the list; it just copies the references to them.
I had a moment of uncertainty about immutable objects; but I checked and indeed, only the reference is copied:
>>> a = [1, 2, 3]
>>> b = a[1:3]
>>> id(a[1])
143727964
>>> id(b[0])
143727964


Input:
Why is equivalent Python code so much slower can somebody explain why is the following trivial code (implementation of Euclid's algorithm to find greatest common denominator) about 3 times slower then equivalent code in Ruby ?
contents of iter_gcd.py:
from sys import argv,stderr

def gcd(m, n):
    if n > m:
        m, n = n, m
    while n != 0:
        rem = m % n
        m = n
        n = rem
    return m

# in Python3 code there is xrange replaced with range function
def main(a1, a2):
    comp = 0
    for j in xrange(a1, 1, -1):
        for i in xrange(1, a2):
            comp += gcd(i,j)

    print(comp)

if __name__ == '__main__':
    if len(argv) != 3:
        stderr.write('usage: {0:s} num1 num2\n'.format(argv[0]))
        exit(1)
    else:
        main(int(argv[1]), int(argv[2]))

contents of iter_gcd.rb:
def gcd(m, n)
    while n != 0
        rem = m % n
        m = n
        n = rem
    end
    return m
end

def main(a1, a2)
    comp = 0
    a1.downto 2 do
        |j|
        1.upto (a2 - 1) do
            |i|
            comp += gcd(i,j)
        end
    end
    puts comp
end

 if __FILE__ == $0
    if ARGV.length != 2
        $stderr.puts('usage: %s num1 num2' % $0)
        exit(1)
    else
        main(ARGV[0].to_i, ARGV[1].to_i)
    end
end

Execution times measurements:
$ time python iter_gcd.py 4000 3000
61356305

real    0m22.890s
user    0m22.867s
sys     0m0.006s

$ python -V
Python 2.6.4

$ time python3 iter_gcd.py 4000 3000
61356305

real    0m18.634s
user    0m18.615s
sys     0m0.009s

$ python3 -V
Python 3.1.2

$ time ruby iter_gcd.rb 4000 3000
61356305

real    0m7.619s
user    0m7.616s
sys     0m0.003s

$ ruby -v
ruby 1.9.2p0 (2010-08-18 revision 29036) [x86_64-linux]

Just curious why I got such results. I considered CPython to be faster in most cases then MRI and even the new Ruby 1.9 on YARV but this "microbenchmark" did really surprised me.
Btw, I know I can use specialised library function like fractions.gcd but I'd like to compare implementations of such basic and trivial language constructs.
Did I miss something or is the implementation of the next Ruby generation so much improved in area of sheer speed ?

Output:
Summary
"Because the function call overhead in Python is much larger than in Ruby."
Details
Being a microbenchmark, this really doesn't say much about the performance of either language in proper use. Likely you would want to rewrite the program to take advantage of the strengths of Python and Ruby, but this does illustrate one of the weak points of Python at the moment. The root cause of the speed differences come from function call overhead. I made a few tests to illustrate. See below for code and more details. For the Python tests, I used 2000 for both gcd parameters.
Interpreter: Python 2.6.6
Program type: gcd using function call
Total CPU time: 29.336 seconds

Interpreter: Python 2.6.6
Program type: gcd using inline code
Total CPU time: 13.194 seconds

Interpreter: Python 2.6.6
Program type: gcd using inline code, with dummy function call
Total CPU  time: 30.672 seconds

This tells us that it's not the calculation made by the gcd function that contributes most to the time difference, it's the function call itself. With Python 3.1, the difference is similar:
Interpreter: Python 3.1.3rc1
Program type: gcd using function call
Total CPU time: 30.920 seconds

Interpreter: Python 3.1.3rc1
Program type: gcd using inline code
Total CPU time: 15.185 seconds

Interpreter: Python 3.1.3rc1
Program type: gcd using inline code, with dummy function call
Total CPU time: 33.739 seconds

Again, the actual calculation is not biggest contributor, it's the function call itself. In Ruby, the function call overhead is much smaller. (Note: I had to use smaller parameters (200) for the Ruby version of the programs because the Ruby profiler really slows down real-time performance. That doesn't affect CPU time performance, though.)
Interpreter: ruby 1.9.2p0 (2010-08-18 revision 29036) [i486-linux]
Program type: gcd using function call
Total CPU time: 21.66 seconds

Interpreter: ruby 1.9.2p0 (2010-08-18 revision 29036) [i486-linux]
Program type: gcd using inline code
Total CPU time: 21.31 seconds

Interpreter: ruby 1.8.7 (2010-08-16 patchlevel 302) [i486-linux]
Program type: gcd using function call
Total CPU time: 27.00 seconds

Interpreter: ruby 1.8.7 (2010-08-16 patchlevel 302) [i486-linux]
Program type: gcd using inline code
Total CPU time: 24.83 seconds

Notice how neither Ruby 1.8 nor 1.9 suffer greatly from the gcd function call â the function call vs. inline version are more or less equal. Ruby 1.9 seems to be a little better with less difference between the function call and inline versions.
So the answer to the question is: "because the function call overhead in Python is much larger than in Ruby".
Code
# iter_gcd -- Python 2.x version, with gcd function call
#             Python 3.x version uses range instead of xrange
from sys import argv,stderr

def gcd(m, n):
    if n > m:
        m, n = n, m
    while n != 0:
        rem = m % n
        m = n
        n = rem
    return m

def main(a1, a2):
    comp = 0
    for j in xrange(a1, 1, -1):
        for i in xrange(1, a2):
            comp += gcd(i,j)
    print(comp)

if __name__ == '__main__':
    if len(argv) != 3:
        stderr.write('usage: {0:s} num1 num2\n'.format(argv[0]))
        exit(1)
    else:
        main(int(argv[1]), int(argv[2]))

# iter_gcd -- Python 2.x version, inline calculation
#             Python 3.x version uses range instead of xrange
from sys import argv,stderr

def main(a1, a2):
    comp = 0
    for j in xrange(a1, 1, -1):
        for i in xrange(1, a2):
            if i < j:
                m, n = j, i
            else:
                m, n = i, j
            while n != 0:
                rem = m % n
                m = n
                n = rem
            comp += m
    print(comp)

if __name__ == '__main__':
    if len(argv) != 3:
        stderr.write('usage: {0:s} num1 num2\n'.format(argv[0]))
        exit(1)
    else:
        main(int(argv[1]), int(argv[2]))

# iter_gcd -- Python 2.x version, inline calculation, dummy function call
#             Python 3.x version uses range instead of xrange
from sys import argv,stderr

def dummyfunc(n, m):
    a = n + m

def main(a1, a2):
    comp = 0
    for j in xrange(a1, 1, -1):
        for i in xrange(1, a2):
            if i < j:
                m, n = j, i
            else:
                m, n = i, j
            while n != 0:
                rem = m % n
                m = n
                n = rem
            comp += m
            dummyfunc(i, j)
    print(comp)

if __name__ == '__main__':
    if len(argv) != 3:
        stderr.write('usage: {0:s} num1 num2\n'.format(argv[0]))
        exit(1)
    else:
        main(int(argv[1]), int(argv[2]))

# iter_gcd -- Ruby version, with gcd function call

def gcd(m, n)
    if n > m
        m, n = n, m
    end
    while n != 0
        rem = m % n
        m = n
        n = rem
    end
    return m
end

def main(a1, a2)
    comp = 0
    a1.downto 2 do
        |j|
        1.upto a2-1 do
            |i|
            comp += gcd(i,j)
        end
    end
    puts comp
end

 if __FILE__ == $0
    if ARGV.length != 2
        $stderr.puts('usage: %s num1 num2' % $0)
        exit(1)
    else
        main(ARGV[0].to_i, ARGV[1].to_i)
    end
end

# iter_gcd -- Ruby version, with inline gcd

def main(a1, a2)
    comp = 0
    a1.downto 2 do |j|
        1.upto a2-1 do |i|
            m, n = i, j
            if n > m
                m, n = n, m
            end
            while n != 0
                rem = m % n
                m = n
                n = rem
            end
            comp += m
        end
    end
    puts comp
end

 if __FILE__ == $0
    if ARGV.length != 2
        $stderr.puts('usage: %s num1 num2' % $0)
        exit(1)
    else
        main(ARGV[0].to_i, ARGV[1].to_i)
    end
end

Test runs
Finally, the commands used to run Python and Ruby with profiling to get the numbers for comparison were pythonX.X -m cProfile iter_gcdX.py 2000 2000 for Python and rubyX.X -rprofile iter_gcdX.rb 200 200 for Ruby. The reason for the difference is that the Ruby profiler adds a lot of overhead. The results are still valid because I'm comparing the difference between a function call and inline code, not the difference between Python and Ruby as such.
See also
Why is python slower compared to Ruby even with this very simple âtestâ?
Is there something wrong with this python code, why does it run so slow compared to ruby?
The Computer Language Benchmarks Game
Google Search: ruby python function call faster


Input:
error: Setup script exited with error: command 'gcc' failed with exit status 1 I get the following error when I try to install MySQL-python-1.2.3 under Python 2.6 in Fedora 14.
Fedora 14 comes with Python 2.7 by default and I am working in a project which runs in Python 2.6, So I am not in a position to update Python from 2.6 to 2.7.
_mysql.c:35:23: fatal error: my_config.h: No such file or directory
compilation terminated.
error: command 'gcc' failed with exit status 1

The complete error message is as below
[root@localhost MySQL-python-1.2.2]# python setup.py build
running build
running build_py
creating build
creating build/lib.linux-i686-2.6
copying _mysql_exceptions.py -> build/lib.linux-i686-2.6
creating build/lib.linux-i686-2.6/MySQLdb
copying MySQLdb/__init__.py -> build/lib.linux-i686-2.6/MySQLdb
copying MySQLdb/converters.py -> build/lib.linux-i686-2.6/MySQLdb
copying MySQLdb/connections.py -> build/lib.linux-i686-2.6/MySQLdb
copying MySQLdb/cursors.py -> build/lib.linux-i686-2.6/MySQLdb
copying MySQLdb/release.py -> build/lib.linux-i686-2.6/MySQLdb
copying MySQLdb/times.py -> build/lib.linux-i686-2.6/MySQLdb
creating build/lib.linux-i686-2.6/MySQLdb/constants
copying MySQLdb/constants/__init__.py -> build/lib.linux-i686-2.6/MySQLdb/constants
copying MySQLdb/constants/CR.py -> build/lib.linux-i686-2.6/MySQLdb/constants
copying MySQLdb/constants/FIELD_TYPE.py -> build/lib.linux-i686-2.6/MySQLdb/constants
copying MySQLdb/constants/ER.py -> build/lib.linux-i686-2.6/MySQLdb/constants
copying MySQLdb/constants/FLAG.py -> build/lib.linux-i686-2.6/MySQLdb/constants
copying MySQLdb/constants/REFRESH.py -> build/lib.linux-i686-2.6/MySQLdb/constants
copying MySQLdb/constants/CLIENT.py -> build/lib.linux-i686-2.6/MySQLdb/constants
running build_ext
building '_mysql' extension
creating build/temp.linux-i686-2.6
gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -Dversion_info=(1,2,2,'final',0) -D__version__=1.2.2 -I/usr/include/mysql -I/usr/local/include/python2.6 -c _mysql.c -o build/temp.linux-i686-2.6/_mysql.o -g -pipe -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m32 -fasynchronous-unwind-tables -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -fno-strict-aliasing -fwrapv -fPIC -DUNIV_LINUX
_mysql.c:35:23: fatal error: my_config.h: No such file or directory
compilation terminated.
error: command 'gcc' failed with exit status 1
[root@localhost MySQL-python-1.2.2]# python2.6 setup.py build
running build
running build_py
copying MySQLdb/release.py -> build/lib.linux-i686-2.6/MySQLdb
running build_ext
building '_mysql' extension
gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -Dversion_info=(1,2,2,'final',0) -D__version__=1.2.2 -I/usr/include/mysql -I/usr/local/include/python2.6 -c _mysql.c -o build/temp.linux-i686-2.6/_mysql.o -g -pipe -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m32 -fasynchronous-unwind-tables -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -fno-strict-aliasing -fwrapv -fPIC -DUNIV_LINUX
_mysql.c:35:23: fatal error: my_config.h: No such file or directory
compilation terminated.
error: command 'gcc' failed with exit status 1

Output:
you need to install MySQL Development package
yum install mysql-devel

:D


Input: How to detect if numpy is installed I'm writing Python code. I want to check if numpy and wxpython are installed on machine. How to do that??

Output:
You can try importing them and then handle the ImportError if the module doesn't exist.
try:
    import numpy
except ImportError:
    print "numpy is not installed"


Input:
How do I use matplotlib autopct? I'd like to create a matplotlib pie chart which has the value of each wedge written on top of the wedge.
The documentation suggests I should use autopct to do this.

autopct: [ None | format string |
  format function ]
      If not None, is a string or function used to label the wedges with
  their numeric value. The label will be
  placed inside the wedge. If it is a
  format string, the label will be
  fmt%pct. If it is a function, it will
  be called.

Unfortunately, I'm unsure what this format string or format function is supposed to be.
Using this basic example below, how can I display each numerical value on top of its wedge?
plt.figure()
values = [3, 12, 5, 8]
labels = ['a', 'b', 'c', 'd']
plt.pie(values, labels=labels) #autopct??
plt.show()

Output:
autopct enables you to display the percent value using Python string formatting. For example, if autopct='%.2f', then for each pie wedge, the format string is '%.2f' and the numerical percent value for that wedge is pct, so the wedge label is set to the string '%.2f'%pct.
import matplotlib.pyplot as plt
plt.figure()
values = [3, 12, 5, 8]
labels = ['a', 'b', 'c', 'd']
plt.pie(values, labels=labels, autopct='%.2f')
plt.show()

yields

You can do fancier things by supplying a callable to autopct. To display both the percent value and the original value, you could do this:
import matplotlib.pyplot as plt

# make the pie circular by setting the aspect ratio to 1
plt.figure(figsize=plt.figaspect(1))
values = [3, 12, 5, 8]
labels = ['a', 'b', 'c', 'd']

def make_autopct(values):
    def my_autopct(pct):
        total = sum(values)
        val = int(round(pct*total/100.0))
        return '{p:.2f}%  ({v:d})'.format(p=pct,v=val)
    return my_autopct

plt.pie(values, labels=labels, autopct=make_autopct(values))
plt.show()

Again, for each pie wedge, matplotlib supplies the percent value pct as the argument, though this time it is sent as the argument to the function my_autopct. The wedge label is set to my_autopct(pct).


Input:
Accessing module level variables, from within a function in the module I'd like to be able to do something like this:
#mymodule
var = None

def load():
    var = something()

Other module(s):
#secondmodule
import mymodule
mymodule.load()

#thirdmodule
from mymodule import var
print var #Shouldn't be None

But I don't know how to reference a modules variable from a function defined in the module.
Is this possible? Or am I going to need to put a global declaration in every place I wan't to use this.
Or am I going at this completely wrong?

Output:
Just change
def load():
    global var
    var = something()

Global variables are read-only from sibling methods. More accurately unless a variable is specified as global, Python consider it as local, but a read access to a local variable name will reach module-level scope if the name is not present in local scope.
See also use of âglobalâ keyword in python and the doc for more details about the global statement


Input:
defaultdict(None) I wish to have a dictionary which contains a set of state transitions. I presumed that I could do this using states = defaultdict(None), but its not working as I expected. For example:
states = defaultdict(None)
if new_state_1 != states["State 1"]:
    dispatch_transition()

I would have thought that states["State 1"] would return the value None and that if new_state is a bool that I would have gotten False for new_state != states["State 1"], but instead I get a KeyError.
What am i doing wrong?
Thanks,
Barry

Output:
defaultdict requires a callable as argument that provides the default-value when invoked without arguments. None is not callable. What you want is this:
defaultdict(lambda: None)


Input:
Determining how many times a substring occurs in a string in Python I am trying to figure out how many times a string occurs in a string. For example:
nStr = '000123000123'

Say the string I want to find is 123. Obviously it occurs twice in nStr but I am having trouble implementing this logic into Python. What I have got at the moment:
pattern = '123'
count = a = 0
while pattern in nStr[a:]:
    a = nStr[a:].find(pattern)+1
    count += 1
return count

The answer it should return is 2. I'm stuck in an infinite loop at the moment.
I was just made aware that count is a much better way to do it but out of curiosity, does anyone see a way to do it similar to what I have already got?

Output:
Use str.count:
>>> nStr = '000123000123'
>>> nStr.count('123')
2

A working version of your code:
nStr = '000123000123'
pattern = '123'
count =0
flag=True
start=0
while flag:
    a = nStr.find(pattern,start)  # find() returns -1 if the word is not found,
                                  #start i the starting index from the search starts(default value is 0)
    if a==-1:          #if pattern not found set flag to False
        flag=False
    else:               # if word is found increase count and set starting index to a+1
        count+=1
        start=a+1
print(count)


Input:
GAE SDK 1.7.4 and InvalidCertificateException Recently, I upgraded my GAE SDK to ver. 1.7.4 and it started to throw 'InvalidCertificateException' when I try to run development server. I searched about this error and some people said it goes away with time, but mine didn't. What should I look into to fix this problem? I am using python framework Django for my app if that has to matter somehow.
$ dev_appserver.py ./
INFO     2012-12-16 07:44:31,412 appcfg.py:586] Checking for updates to the SDK.
Traceback (most recent call last):
  File "/usr/local/bin/dev_appserver.py", line 171, in <module>
    run_file(__file__, globals())
  File "/usr/local/bin/dev_appserver.py", line 167, in run_file
    execfile(script_path, globals_)
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/dev_appserver_main.py", line 747, in <module>
    sys.exit(main(sys.argv))
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/dev_appserver_main.py", line 680, in main
    update_check.CheckForUpdates()
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/appcfg.py", line 597, in CheckForUpdates
    runtime=self.config.runtime)
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/appengine_rpc.py", line 391, in Send
    f = self.opener.open(req)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 394, in open
    response = self._open(req, data)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 412, in _open
    '_open', req)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 372, in _call_chain
    result = func(*args)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1207, in https_open
    return self.do_open(httplib.HTTPSConnection, req)
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/fancy_urllib/fancy_urllib/__init__.py", line 379, in do_open
    url_error.reason.args[1])
fancy_urllib.InvalidCertificateException: Host appengine.google.com returned an invalid certificate (_ssl.c:503: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed):
To learn more, see http://code.google.com/appengine/kb/general.html#rpcssl

Output:
Quick workaround that I found: delete the file google_appengine/lib/cacerts/cacerts.txt from your installed SDK.
Starting from the GoogleAppEngineLauncher:

GoogleAppEngineLauncher/Contents/Resources/GoogleAppEngineDefault.bundle/Contentââs/Resources/google_appengine/lib/cacerts/cacerts.txt

EDIT #

as of google app engine SDK 1.8.1 this file as been renamed to
  urlfetch_cacerts.txt. Still in the same directory and removing it
  still fixes the problem.

â @Harrison


Input:
How to flatten only some dimensions of a numpy array Is there a quick way to "sub-flatten" or flatten only some of the first dimensions in a numpy array?
For example, given a numpy array of dimensions (50,100,25), the resultant dimensions would be (5000,25)
Thanks

Output:
A slight generalization to Alexander's answer - np.reshape can take -1 as an argument, meaning "total array size divided by product of all other listed dimensions":
e.g. to flatten all but the last dimension:
>>> arr = numpy.zeros((50,100,25))
>>> new_arr = arr.reshape(-1, arr.shape[-1])
>>> new_arr.shape
# (5000, 25)


Input:
What is the difference between `sorted(list)` vs `list.sort()` ? python list.sort() sorts the list and save the sorted list, while sorted(list) returns a sorted list without changing the original list.

But when to use which?
And which is faster? And how much faster?
Can a list's original positions be retrieved after list.sort()?

Output:
sorted() returns a new sorted list, leaving the original list unaffected. list.sort() sorts the list in-place, mutating the list indices, and returns None (like all in-place operations).
sorted() works on any iterable, not just lists. Strings, tuples, dictionaries (you'll get the keys), generators, etc., returning a list containing all elements, sorted.

Use list.sort() when you want to mutate the list, sorted() when you want a new sorted object back. Use sorted() when you want to sort something that is an iterable, not a list yet.
For lists, list.sort() is faster than sorted() because it doesn't have to create a copy. For any other iterable, you have no choice.
No, you cannot retrieve the original positions. Once you called list.sort() the original order is gone.


Input:
Quicksort sorts larger numbers faster? I was messing around with Python trying to practice my sorting algorithms and found out something interesting.
I have three different pieces of data:

x = number of numbers to sort
y = range the numbers are in (all random generated ints)
z = total time taken to sort

When:
x = 100000 and
y = (0,100000) then
z =  0.94182094911 sec
When:
x = 100000 and
y = (0,100) then
z = 12.4218382537 sec
When:
x = 100000 and
y = (0,10) then
z =  110.267447809 sec
Any ideas?
Code:
import time
import random
import sys

#-----Function definitions

def quickSort(array): #random pivot location quicksort. uses extra memory.
    smaller = []
    greater = []
    if len(array) <= 1:
        return array
    pivotVal = array[random.randint(0, len(array)-1)]
    array.remove(pivotVal)
    for items in array:
        if items <= pivotVal:
            smaller.append(items)
        else:
            greater.append(items)
    return concat(quickSort(smaller), pivotVal, quickSort(greater))

def concat(before, pivot, after):
    new = []
    for items in before:
        new.append(items)
    new.append(pivot)
    for things in after:
        new.append(things)
    return new

#-----Variable definitions
list = []
iter = 0
sys.setrecursionlimit(20000)
start = time.clock() #start the clock

#-----Generate the list of numbers to sort
while(iter < 100000):
    list.append(random.randint(0,10))  #modify this to change sorting speed
    iter = iter + 1
timetogenerate = time.clock() - start #current timer - last timer snapshot

#-----Sort the list of numbers
list = quickSort(list)
timetosort = time.clock() - timetogenerate #current timer - last timer snapshot

#-----Write the list of numbers
file = open("C:\output.txt", 'w')
for items in list:
    file.write(str(items))
    file.write("\n")
file.close()
timetowrite = time.clock() - timetosort #current timer - last timer snapshot

#-----Print info
print "time to start: " + str(start)
print "time to generate: " + str(timetogenerate)
print "time to sort: " + str(timetosort)
print "time to write: " + str(timetowrite)
totaltime = timetogenerate + timetosort + start
print "total time: " + str(totaltime)

-------------------revised NEW code----------------------------
def quickSort(array): #random pivot location quicksort. uses extra memory.
    smaller = []
    greater = []
    equal = []
    if len(array) <= 1:
        return array
    pivotVal = array[random.randint(0, len(array)-1)]
    array.remove(pivotVal)
    equal.append(pivotVal)
    for items in array:
        if items < pivotVal:
            smaller.append(items)
        elif items > pivotVal:
            greater.append(items)
        else:
            equal.append(items)
    return concat(quickSort(smaller), equal, quickSort(greater))

def concat(before, equal, after):
    new = []
    for items in before:
        new.append(items)
    for items in equal:
        new.append(items)
    for items in after:
        new.append(items)
    return new

Output:
I think this has to do with the choice of a pivot.  Depending on how your partition step works, if you have a lot of duplicate values, your algorithm can degenerate to quadratic behavior when confronted with many duplicates.  For example, suppose that you're trying to quicksort this stream:
 [0 0 0 0 0 0 0 0 0 0 0 0 0]

If you aren't careful with how you do the partitioning step, this can degenerate quickly.  For example, suppose you pick your pivot as the first 0, leaving you with the array
 [0 0 0 0 0 0 0 0 0 0 0 0]

to partition.  Your algorithm might say that the smaller values are the array
 [0 0 0 0 0 0 0 0 0 0 0 0]

And the larger values are the array
 []

This is the case that causes quicksort to degenerate to O(n2), since each recursive call is only shrinking the size of the input by one (namely, by pulling off the pivot element).
I noticed that in your code, your partitioning step does indeed do this:
for items in array:
    if items <= pivotVal:
        smaller.append(items)
    else:
        greater.append(items)

Given a stream that's a whole bunch of copies of the same element, this will put all of them into one array to recursively sort.
Of course, this seems like a ridiculous case - how is this at all connected to reducing the number of values in the array? - but it actually does come up when you're sorting lots of elements that aren't distinct.  In particular, after a few passes of the partitioning, you're likely to group together all equal elements, which will bring you into this case.
For a discussion of how to prevent this from happening, there's a really great talk by Bob Sedgewick and Jon Bentley about how to modify the partition step to work quickly when in the presence of duplicate elements.  It's connected to Dijkstra's Dutch national flag problem, and their solutions are really clever.
One option that works is to partition the input into three groups - less, equal, and greater.  Once you've broken the input up this way, you only need to sort the less and greater groups; the equal groups are already sorted.  The above link to the talk shows how to do this more or less in-place, but since you're already using an out-of-place quicksort the fix should be easy.  Here's my attempt at it:
for items in array:
    if items < pivotVal:
        smaller.append(items)
    elif items == pivotVal:
        equal.append(items)
    else:
        greater.append(items)

I've never written a line of Python in my life, BTW, so this may be totally illegal syntax.  But I hope the idea is clear!  :-)


Input:
Python IndentationError: unexpected indent I really can't see the indentation error here, i'm getting crazy with this ->
# loop
while d <= end_date:
    # print d.strftime("%Y%m%d")
    fecha = d.strftime("%Y%m%d")
    # set url
    url = 'http://www.wpemergencia.omie.es//datosPub/marginalpdbc/marginalpdbc_' + fecha + '.1'
    # Descargamos fichero
    response = urllib2.urlopen(url)
    # Abrimos fichero
    output = open(fname,'wb')
    # Escribimos fichero
    output.write(response.read())
    # Cerramos y guardamos fichero
    output.close()
    # fecha++
    d += delta

Output:
Run your program with
python -t script.py

This will warn you if you have mixed tabs and spaces.
On *nix systems, you can see where the tabs are by running
cat -A script.py

and you can automatically convert tabs to 4 spaces with the command
expand -t 4 script.py > fixed_script.py

PS. Be sure to use a programming editor (e.g. emacs, vim), not a word processor, when programming. You won't get this problem with a programming editor.
PPS. For emacs users, M-x whitespace-mode will show the same info as cat -A from within an emacs buffer!


Input:
Passing STATIC_URL to file javascript with django Which is the best solution to pass {{ STATIC_URL }} to javascript files?
I'm working with django and python.
Thanks in advance.
Regards.

Output:
Using a global javascript variable with the static url value is more simple :
<script language="javascript">var STATIC_URL = "{{ STATIC_URL|escapejs }}";</script>
<script src="{{ STATIC_URL }}js/myfile.js"></script>

Then, you can simply use the static url by calling STATIC_URL in myfile.js :
html = '<img src="'+STATIC_URL+'/icons/flags/tn.gif">';


Input:
django-debug-toolbar breaking on admin while getting sql stats Environment:django debug toolbar breaking while using to get sql stats else it's working fine on the other pages, breaking only on the pages which have sql queries.
Request Method: GET
Request URL: http://www.blog.local/admin/

Django Version: 1.9.7
Python Version: 2.7.6
Installed Applications:
[
 ....
 'django.contrib.staticfiles',
 'debug_toolbar']
Installed Middleware:
[
  ...
 'debug_toolbar.middleware.DebugToolbarMiddleware']

Traceback:

File "/home/vagrant/www/dx/venv/local/lib/python2.7/site-packages/django/core/handlers/base.py" in get_response
  235.                 response = middleware_method(request, response)

File "/home/vagrant/www/dx/venv/local/lib/python2.7/site-packages/debug_toolbar/middleware.py" in process_response
  129.                 panel.generate_stats(request, response)

File "/home/vagrant/www/dx/venv/local/lib/python2.7/site-packages/debug_toolbar/panels/sql/panel.py" in generate_stats
  192.                     query['sql'] = reformat_sql(query['sql'])

File "/home/vagrant/www/dx/venv/local/lib/python2.7/site-packages/debug_toolbar/panels/sql/utils.py" in reformat_sql
  27.     return swap_fields(''.join(stack.run(sql)))

File "/home/vagrant/www/dx/venv/local/lib/python2.7/site-packages/sqlparse/engine/filter_stack.py" in run
  29.             stream = filter_.process(stream)

Exception Type: TypeError at /admin/
 Exception Value: process() takes exactly 3 arguments (2 given)

Output:
sqlparse latest version was released today and it's not compatible with django-debug-toolbar version 1.4, Django version 1.9
workaround is force pip to install sqlparse==0.1.19


Input:
Algorithm for neatly indenting SQL statements (Python implementation would be nice) I'd like to reformat some SQL statements that are a single string with newlines in to something that's much easier to read.
I don't personally know of a good coding style for indenting SQL - how should nested queries / where clauses / left joins / etc by represented to maximise readability?
Has anyone seen a pretty-printing algorithm that does this already? In Python would be even better.

Output: You can try sqlparse. It's a Python module that provides simple SQL formatting. A online demo is available here.


Input: Python way to clone a git repository Is there a Python way without using a subprocess to clone a git repository? I'm up for using any sort of modules you recommend.

Output:
Using GitPython will give you a good python interface to Git.
For example, after installing it (pip install gitpython), for cloning a new repository you can use clone_from function:
from git import Repo

Repo.clone_from(git_url, repo_dir)

Note: GitPython requires git being installed on the system, and accessible via system's PATH.


Input:
What is a good way to do countif in Python I want to count how many members of an iterable meet a given condition. I'd like to do it in a way that is clear and simple and preferably reasonably optimal.
My current best ideas are:
sum(meets_condition(x) for x in my_list)

and
len([x for x in my_list if meets_condition(x)])

The first one being iterator based is presumably faster for big lists. And it's the same form as you'd use for testing any and all. However it depends on the fact that int(True) == 1, which is somewhat ugly.
The second one seems easier to read to me, but it is different from the any and all forms.
Does anyone have any better suggestions? is there a library function somewhere that I am missing?

Output:
The iterator based approach is just fine. There are some slight modifications that can emphasize the fact that you are counting:
sum(1 if meets_condition(x) else 0 for x in my_list)
# or
sum(1 for x in my_list if meets_condition(x))

And as always, if the intent isn't apparent from the code, encapsulate it in descriptively named function:
def count_matching(condition, seq):
    """Returns the amount of items in seq that return true from condition"""
    return sum(1 for item in seq if condition(item))

count_matching(meets_condition, my_list)


Input:
How to keep all my django applications in specific folder I have a Django project, let's say "project1".
Typical folder structure for applications is:
/project1/
         /app1/
         /app2/
         ...
         __init__.py
         manage.py
         settings.py
         urls.py

What should I do if I want to hold all of my applications in some separate folder, 'apps' for example? So that structure should look like the following:
/project/
         apps/
              app1/
              app2/
              ...
         __init__.py
         manage.py
         settings.py
         urls.py

Output:
You can add your apps folder to your python path by inserting the following in your settings.py:
import os
import sys

PROJECT_ROOT = os.path.dirname(__file__)
sys.path.insert(0, os.path.join(PROJECT_ROOT, 'apps'))

Then you can use all the apps in this folder just in the same way as they were in your project root!


Input:
Import from sibling directory I have a Python class called "ClassA" and another Python class which is supposed to import ClassA which is "ClassB". The directory structure is as follows:
MainDir
../Dir
..../DirA/ClassA
..../DirB/ClassB

How would I use sys.path so that ClassB can use ClassA?

Output:
as a literal answer to the question 'Python Import from parent directory':
to import 'mymodule' that is in the parent directory of your current module:
import os
parentdir = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
os.sys.path.insert(0,parentdir)
import mymodule

edit
Unfortunately, the __file__ attribute is not always set.
A more secure way to get the parentdir is through the inspect module:
import inspect
currentdir = os.path.dirname(os.path.abspath(inspect.getfile(inspect.currentframe())))
parentdir = os.path.dirname(currentdir)


Input:
How can I read the contents of an URL with Python? The following works when I paste it on the browser:
http://www.somesite.com/details.pl?urn=2344

But when I try reading the URL with Python nothing happens:
 link = 'http://www.somesite.com/details.pl?urn=2344'
 f = urllib.urlopen(link)
 myfile = f.readline()
 print myfile

Do I need to encode the URL, or is there something I'm not seeing?

Output:
To answer your question:
import urllib

link = "http://www.somesite.com/details.pl?urn=2344"
f = urllib.urlopen(link)
myfile = f.read()
print myfile

You need to read(), not readline()
Or, just get this library here: http://docs.python-requests.org/en/latest/ and seriously use it :)
import requests

link = "http://www.somesite.com/details.pl?urn=2344"
f = requests.get(link)

print f.text


Input:
Pandas Combining 2 Data Frames (join on a common column) I have 2 dataframes:
restaurant_ids_dataframe
Data columns (total 13 columns):
business_id      4503  non-null values
categories       4503  non-null values
city             4503  non-null values
full_address     4503  non-null values
latitude         4503  non-null values
longitude        4503  non-null values
name             4503  non-null values
neighborhoods    4503  non-null values
open             4503  non-null values
review_count     4503  non-null values
stars            4503  non-null values
state            4503  non-null values
type             4503  non-null values
dtypes: bool(1), float64(3), int64(1), object(8)`

and
restaurant_review_frame
Int64Index: 158430 entries, 0 to 229905
Data columns (total 8 columns):
business_id    158430  non-null values
date           158430  non-null values
review_id      158430  non-null values
stars          158430  non-null values
text           158430  non-null values
type           158430  non-null values
user_id        158430  non-null values
votes          158430  non-null values
dtypes: int64(1), object(7)

I would like to join these two DataFrames to make them into a single dataframe using the DataFrame.join() command in pandas.
I have tried the following line of code:
#the following line of code creates a left join of restaurant_ids_frame and   restaurant_review_frame on the column 'business_id'
restaurant_review_frame.join(other=restaurant_ids_dataframe,on='business_id',how='left')

But when I try this I get the following error:
Exception: columns overlap: Index([business_id, stars, type], dtype=object)

I am very new to pandas and have no clue what I am doing wrong as far as executing the join statement is concerned.
any help would be much appreciated.

Output:
You can use merge to combine two dataframes into one:
import pandas as pd
pd.merge(restaurant_ids_dataframe, restaurant_review_frame, on='business_id', how='outer')

where on specifies field name that exists in both dataframes to join on, and how
defines whether its inner/outer/left/right join, with outer using 'union of keys from both frames (SQL: full outer join).' Since you have 'star' column in both dataframes, this by default will create two columns star_x and star_y in the combined dataframe. As @DanAllan mentioned for the join method, you can modify the suffixes for merge by passing it as a kwarg. Default is suffixes=('_x', '_y'). if you wanted to do something like star_restaurant_id and star_restaurant_review, you can do:
 pd.merge(restaurant_ids_dataframe, restaurant_review_frame, on='business_id', how='outer', suffixes=('_restaurant_id', '_restaurant_review'))

The parameters are explained in detail in this link.


Input:
Is it possible to generate and return a ZIP file with App Engine? I have a small project that would be perfect for Google App Engine.  Implementing it hinges on the ability to generate a ZIP file and return it.
Due to the distributed nature of App Engine, from what I can tell, the ZIP file couldn't be created "in-memory" in the traditional sense.  It would basically have to be generated and and sent in a single request/response cycle.
Does the Python zip module even exist in the App Engine environment?

Output:
zipfile is available at appengine and reworked example follows:
from contextlib import closing
from zipfile import ZipFile, ZIP_DEFLATED

from google.appengine.ext import webapp
from google.appengine.api import urlfetch

def addResource(zfile, url, fname):
    # get the contents
    contents = urlfetch.fetch(url).content
    # write the contents to the zip file
    zfile.writestr(fname, contents)

class OutZipfile(webapp.RequestHandler):
    def get(self):
        # Set up headers for browser to correctly recognize ZIP file
        self.response.headers['Content-Type'] ='application/zip'
        self.response.headers['Content-Disposition'] = \
            'attachment; filename="outfile.zip"'

        # compress files and emit them directly to HTTP response stream
        with closing(ZipFile(self.response.out, "w", ZIP_DEFLATED)) as outfile:
            # repeat this for every URL that should be added to the zipfile
            addResource(outfile,
                'https://www.google.com/intl/en/policies/privacy/',
                'privacy.html')
            addResource(outfile,
                'https://www.google.com/intl/en/policies/terms/',
                'terms.html')


Input:
How to create a UserProfile form in Django with first_name, last_name modifications? If think my question is pretty obvious and almost every developer working with UserProfile should be able to answer it.
However, I could not find any help on the django documentation or in the Django Book.
When you want to do a UserProfile form in with Django Forms, you'd like to modify the profile fields as well as some User field.
But there is no forms.UserProfileForm (yet?) !
How do you do that ?

Output:
Here is how I finally did :
class UserProfileForm(forms.ModelForm):
    first_name = forms.CharField(label=_(u'PrÃ©nom'), max_length=30)
    last_name = forms.CharField(label=_(u'Nom'), max_length=30)

    def __init__(self, *args, **kw):
        super(UserProfileForm, self).__init__(*args, **kw)
        self.fields['first_name'].initial = self.instance.user.first_name
        self.fields['last_name'].initial = self.instance.user.last_name

        self.fields.keyOrder = [
            'first_name',
            'last_name',
            ...some_other...
            ]

    def save(self, *args, **kw):
        super(UserProfileForm, self).save(*args, **kw)
        self.instance.user.first_name = self.cleaned_data.get('first_name')
        self.instance.user.last_name = self.cleaned_data.get('last_name')
        self.instance.user.save()

    class Meta:
        model = UserProfile


Input:
attribute 'tzinfo' of 'datetime.datetime' objects is not writable How do I set the timezone of a datetime instance that just came out of the datastore?
When it first comes out it is in UTC.  I want to change it to EST.
I'm trying, for example:
class Book( db.Model ):
    creationTime = db.DateTimeProperty()

When a Book is retrieved, I want to set its tzinfo immediately:
book.creationTime.tzinfo = EST

Where I use this example for my EST object
However I get:

attribute 'tzinfo' of 'datetime.datetime' objects is not writable

I've seen a number of answers that recommend pytz and python-dateutil, but I really want an answer to this question.

Output:
datetime's objects are immutable, so you never change any of their attributes -- you make a new object with some attributes the same, and some different, and assign it to whatever you need to assign it to.
I.e., in your case, instead of
book.creationTime.tzinfo = EST

you have to code
book.creationTime = book.creationTime.replace(tzinfo=EST)


Input:
The "next" parameter, redirect, django.contrib.auth.login I'm trying to redirect users to custom url "/gallery/(username)/" after successfully logging in.  It currently redirects to the default "/account/profile/"  While I know what I can override the redirect url in my settings.py, my url is dynamic thus it will not work.
Documentation states that I need to use the "next" parameter and context processors.  I have the {{next}} in my template, but I'm confused on how to actually pass the "/gallery/(username)".  Any help would be greatly appreciated.
p.s I'm trying to steer away from writing my own login view.

Output: Django's login view django.contrib.auth.views.login accepts a dictionary named extra_context. The values in the dictionary are directly passed to the template. So you can use that to set the next parameter. Once that is done, you can set a hidden field with name next and value {{ next }} so that it gets rendered in the template.


Input: python count days ignoring weekends how can I calculate number of days between two dates ignoring weekends ?

Output:
>>> from datetime import date,timedelta
>>> fromdate = date(2010,1,1)
>>> todate = date(2010,3,31)
>>> daygenerator = (fromdate + timedelta(x + 1) for x in xrange((todate - fromdate).days))
>>> sum(1 for day in daygenerator if day.weekday() < 5)
63

This creates a generator using a generator expression which will yield the list of days to get from the fromdate to todate.
We could then create a list from the generator, filtering out weekends using the weekday() function, and the size of the list gives the number of days we want.  However, to save having the whole list in memory which could be a problem if the dates are a long time apart we use another generator expression which filters out weekends but returns 1 instead of each date.  We can then just add all these 1s together to get the length without having to store the whole list.
Note, if fromdate == todate this calculate 0 not 1.


Input:
Using a regular expression to replace upper case repeated letters in python with a single lowercase letter I am trying to replace any instances of uppercase letters that repeat themselves twice in a string with a single instance of that letter in a lower case. I am using the following regular expression and it is able to match the repeated upper case letters, but I am unsure as how to make the letter that is being replaced lower case.
import re
s = 'start TT end'
re.sub(r'([A-Z]){2}', r"\1", s)
>>> 'start T end'

How can I make the "\1" lower case? Should I not be using a regular expression to do this?

Output:
Pass a function as the repl argument. The MatchObject is passed to this function and .group(1) gives the first parenthesized subgroup:
import re
s = 'start TT end'
callback = lambda pat: pat.group(1).lower()
re.sub(r'([A-Z]){2}', callback, s)

EDIT
And yes, you should use ([A-Z])\1 instead of ([A-Z]){2} in order to not match e.g. AZ. (See @bobince's answer.)
import re
s = 'start TT end'
re.sub(r'([A-Z])\1', lambda pat: pat.group(1).lower(), s) # Inline

Gives:
'start t end'


Input:
How to create a python 2.x package - simple case Please show the simple and up to date standard way to create a python package for python 2.x
I'd prefer to use pip for installing the package later.
The package should contain a single class:
class hello:
  def greet(self):
    print "hello"

One should be able to do the following later:
pip install my_package-0.1.1....

And then using it:
from my_package import hello

h = hello.hello()
h.greet()

What I am asking for is:

The directory and file layout
Contents of the files
command to create the distributable package file
command to install the package from the distributable package file (using preferably pip)

There are several howtos that I found but I am still not sure how this very simple and stripped down case (no nested packages, removal off all files and features that can be omitted for the most simple case) would be handled and which is the modern way to do it.
I would like this question to enter community wiki state, so you won't get any points and I will give enough time and will mark an answer accepted after several days, also considering the votes and comments.
Edit:
I have a first running example that I want to share, I used Marius Gedminas's answer for it. It does not contain everything that should be there, but it works, so it can demonstrate the core of the technical process. To add more necessary parts please read Marius's answer below.
Directory structure:
MyProject/
    setup.py
    my_package.py
    README.txt
    MANIFEST.in

setup.py:
from setuptools.import setup
setup(name='MyProject',
      version='0.1',
      py_modules=['my_package'])

my_package.py:
class hello:
  def greet(self):
    print "hello"

MANIFEST.in:
include *.txt

To create the package from this folder, go into the folder MyProject and run:
$ python setup.py sdist

This will create a file MyProject-0.1.tar.gz in a subfolder dist/. Copy this file to a folder on the target machine.
On the target machine run this command in the folder containing MyProject-0.1.tar.gz:
sudo pip install MyProject-0.1.tar.gz

It can be necessary to logout and re-login on the target machine now, so the package will be found. Afterwards you can test the package on the target machine using the python shell:
$ python
>>> import my_package
>>> h = my_package.hello()
>>> h.greet()
hello
>>>

Once this works please remember to add the other necessary contents, see Marius's answer below.

Output:
Start simple
Simplest one-file package:
MyProject/
    setup.py
    my_package.py

Simplest setup.py:
from setuptools import setup
setup(name='MyProject',
      version='0.1',
      author='Your Name',
      author_email='your.name@example.com',
      license='MIT',
      description='Example package that says hello',
      py_modules=['my_package'])

Including extra files in the package
Next you should probably add a README:
MyProject/
    MANIFEST.in
    README.rst
    setup.py
    my_package.py

Note the new file -- MANIFEST.in.  It specifies which non-Python files ought to be included in your source distribution:
include *.rst

People will tell you "oh, skip the manifest, just add the files to source control, setuptools will find them".  Ignore that advice, it's too error-prone.
Making the PyPI page useful
It's useful to make the README.rst available for people to view online, on the Python Package Index.  So change your setup.py to do
from setuptools import setup
with open('README.rst') as f:
    readme = f.read()
setup(name='MyProject',
      ...
      description='Example package that says hello',
      long_description=readme,
      ...)

Use ReStructuredText markup for prettier pages.  Use
python setup.py --long-description | rst2html

to catch ReStructuredText errors early.
More than one Python module in a package
One file will not be enough soon, so change it to a package (confusing terminology warning: Python package as in a directory with a __init__ py, not as in a distributable self-contained archive):
MyProject/
    MANIFEST.in
    README.rst
    setup.py
    my_package/
        __init__.py
        some_module.py

and change setup.py to
from setuptools import setup, find_packages
with open('README.rst') as f:
    readme = f.read()
setup(name='MyProject',
      version='0.2',
      author='Your Name',
      author_email='your@email',
      license='MIT',
      description='Example package that says hello',
      long_description=readme,
      packages=find_packages())

Releasing to the public
Get a PyPI account -- you only need to do this once.
To make a release, make sure the version number in setup.py is correct, then run
python setup.py sdist register upload

That's it.
Telling people to install it
Tell them to
pip install MyProject

(same name you specified in setup.py as the name argument to setup())


Input:
How do you use subprocess.check_output() in Python? I have found documentation about subprocess.check_output() but I cannot find one with  arguments and the documentation is not very in depth. I am using Python 3 (but am trying to run a Python 2 file through Python 3)
I am trying to run this command:
python py2.py -i test.txt
-i is a positional argument for argparse, test.txt is what the -i is, py2.py is the file to run
I have tried a lot of (non working) variations including:
py2output = subprocess.check_output([str('python py2.py '),'-i', 'test.txt'])
py2output = subprocess.check_output([str('python'),'py2.py','-i', test.txt'])

Output:
The right answer is:
py2output = subprocess.check_output(['python','py2.py','-i', 'test.txt'])

To demonstrate, here are my two programs:
py2.py:
import sys
print sys.argv

py3.py:
import subprocess
py2output = subprocess.check_output(['python', 'py2.py', '-i', 'test.txt'])
print('py2 said:', py2output)

Running it:
$ python3 py3.py
py2 said: b"['py2.py', '-i', 'test.txt']\n"

Here's what's wrong with each of your versions:
py2output = subprocess.check_output([str('python py2.py '),'-i', 'test.txt'])

First, str('python py2.py') is exactly the same thing as 'python py2.py'âyou're taking a str, and calling str to convert it to an str. This makes the code harder to read, longer, and even slower, without adding any benefit.
More seriously, python py2.py can't be a single argument, unless you're actually trying to run a program named, say, /usr/bin/python\ py2.py. Which you're not; you're trying to run, say, /usr/bin/python with first argument py2.py. So, you need to make them separate elements in the list.
Your second version fixes that, but you're missing the ' before test.txt'. This should give you a SyntaxError, probably saying EOL while scanning string literal.
Meanwhile, I'm not sure how you found documentation but couldn't find any examples with arguments. The very first example is:
>>> subprocess.check_output(["echo", "Hello World!"])
b'Hello World!\n'

That calls the "echo" command with an additional argument, "Hello World!".
Also:

-i is a positional argument for argparse, test.txt is what the -i is

I'm pretty sure -i is not a positional argument, but an optional argument. Otherwise, the second half of the sentence makes no sense.