Input:
Any experiences with Protocol Buffers? I was just looking through some information about Google's protocol buffers data interchange format.  Has anyone played around with the code or even created a project around it?
I'm currently using XML in a Python project for structured content created by hand in a text editor, and I was wondering what the general opinion was on Protocol Buffers as a user-facing input format.  The speed and brevity benefits definitely seem to be there, but there are so many factors when it comes to actually generating and processing the data.

Output:
Protocol buffers are intended to optimize communications between machines. They are really not intended for human interaction. Also, the format is binary, so it could not replace XML in that use case.
I would also recommend JSON as being the most compact text-based format.


Input:
104, 'Connection reset by peer' socket error, or When does closing a socket result in a RST rather than FIN? We're developing a Python web service and a client web site in parallel.  When we make an HTTP request from the client to the service, one call consistently raises a socket.error in socket.py, in read:
(104, 'Connection reset by peer')
When I listen in with wireshark, the "good" and "bad" responses look very similar:

Because of the size of the OAuth header, the request is split into two packets.  The service responds to both with ACK
The service sends the response, one packet per header (HTTP/1.0 200 OK, then the Date header, etc.).  The client responds to each with ACK.
(Good request) the server sends a FIN, ACK.  The client responds with a FIN, ACK.  The server responds ACK.
(Bad request) the server sends a RST, ACK, the client doesn't send a TCP response, the socket.error is raised on the client side.

Both the web service and the client are running on a Gentoo Linux x86-64 box running glibc-2.6.1.  We're using Python 2.5.2 inside the same virtual_env.
The client is a Django 1.0.2 app that is calling httplib2 0.4.0 to make requests.  We're signing requests with the OAuth signing algorithm, with the OAuth token always set to an empty string.
The service is running Werkzeug 0.3.1, which is using Python's wsgiref.simple_server.  I ran the WSGI app through wsgiref.validator with no issues.
It seems like this should be easy to debug, but when I trace through a good request on the service side, it looks just like the bad request, in the socket._socketobject.close() function, turning delegate methods into dummy methods.  When the send or sendto (can't remember which) method is switched off, the FIN or RST is sent, and the client starts processing.
"Connection reset by peer" seems to place blame on the service, but I don't trust httplib2 either.  Can the client be at fault?
** Further debugging - Looks like server on Linux **
I have a MacBook, so I tried running the service on one and the client website on the other.  The Linux client calls the OS X server without the bug (FIN ACK).  The OS X client calls the Linux service with the bug (RST ACK, and a (54, 'Connection reset by peer')).  So, it looks like it's the service running on Linux.  Is it x86_64? A bad glibc? wsgiref? Still looking...
** Further testing - wsgiref looks flaky **
We've gone to production with Apache and mod_wsgi, and the connection resets have gone away.  See my answer below, but my advice is to log the connection reset and retry.  This will let your server run OK in development mode, and solidly in production.

Output:
I've had this problem.  See The Python "Connection Reset By Peer" Problem.
You have (most likely) run afoul of small timing issues based on the Python Global Interpreter Lock.
You can (sometimes) correct this with a time.sleep(0.01) placed strategically.
"Where?" you ask.  Beats me.  The idea is to provide some better thread concurrency in and around the client requests.  Try putting it just before you make the request so that the GIL is reset and the Python interpreter can clear out any pending threads.


Input:
svg diagrams using python I am looking for a library to generate svg diagrams in python (I fetch data from a sql database). I have found python-gd, but it has not much documentation and last update was in 2005 so I wonder if there are any other libraries that are good for this purpose.
I am mostly thinking about simple line graphs, something like this:

Output:
As you're looking for simple line graphics, probably, CairoPlot will fit your needs as it can generate svg output files out of the box. Take a look at this.

This example image shows only a few of its capabilities. Using the trunk version available at launchpad you'll be able to add a legend box and add axis titles.
Besides that, using the trunk version, it's possible to generate:

DotLine charts (the ones I believe you need)
Scatter charts
Pie/Donut charts
Horizontal/Vertical Bar charts
Gantt charts


Input:
Fastest Way to Delete a Line from Large File in Python I am working with a very large (~11GB) text file on a Linux system.  I am running it through a program which is checking the file for errors.  Once an error is found, I need to either fix the line or remove the line entirely.  And then repeat...
Eventually once I'm comfortable with the process, I'll automate it entirely.  For now however, let's assume I'm running this by hand.
What would be the fastest (in terms of execution time) way to remove a specific line from this large file?  I thought of doing it in Python...but would be open to other examples.  The line might be anywhere in the file.
If Python, assume the following interface:
def removeLine(filename, lineno):
Thanks,
-aj

Output:
You can have two file objects for the same file at the same time (one for reading, one for writing):
def removeLine(filename, lineno):
    fro = open(filename, "rb")

    current_line = 0
    while current_line < lineno:
        fro.readline()
        current_line += 1

    seekpoint = fro.tell()
    frw = open(filename, "r+b")
    frw.seek(seekpoint, 0)

    # read the line we want to discard
    fro.readline()

    # now move the rest of the lines in the file
    # one line back
    chars = fro.readline()
    while chars:
        frw.writelines(chars)
        chars = fro.readline()

    fro.close()
    frw.truncate()
    frw.close()


Input:
How to create simple web site with python? How to create simple web site with python?
I mean really simple, f.ex, you see text "Hello World", and there are button "submit", which (onClick) will show ajax box "submit successful".
I want to start develop some stuff with Python, and I don't know where to start ;)

Output:
I was hoping more elaborate answers would be given to this question, since it's a sensitive subject. Python web developing is split across lots of frameworks, each with its ups and downs and every developer using a different one. This is quite unfortunate. What you should know:

Use WSGI. Don't use anything else, WSGI is the latest standard in Python web developing;
Don't develop directly on top of WSGI, unless you really have to (not even for a hello world app);
Use a framework that best suits your needs:

I played with Werkzeug (which is not really a platform, they call it a toolkit) because it's really simple yet powerful. It lets you work on the WSGI level (also helps you understand how WSGI works) while providing really useful features and helpers. On the Werkzeug website you will also find useful tutorials and things like that.
Probably the most popular framework is Django. Never used it, but maybe there's a reason for why it is so popular.

In conclusion, use whatever is closest to your heart.


Input:
Extract images from PDF without resampling, in python? How might one extract all images from a pdf document, at native resolution and format? (Meaning extract tiff as tiff, jpeg as jpeg, etc. and without resampling). Layout is unimportant, I don't care were the source image is located on the page.
I'm using python 2.7 but can use 3.x if required.

Output: Often in a PDF, the image is simply stored as-is.  For example, a PDF with a jpg inserted will have a range of bytes somewhere in the middle that when extracted is a valid jpg file.  You can use this to very simply extract byte ranges from the PDF.  I wrote about this some time ago, with sample code: Extracting JPGs from PDFs.


Input:
Fastest Way to generate 1,000,000+ random numbers in python I am currently writing an app in python that needs to generate large amount of random numbers, FAST. Currently I have a scheme going that uses numpy to generate all of the numbers in a giant batch (about ~500,000 at a time). While this seems to be faster than python's implementation. I still need it to go faster. Any ideas? I'm open to writing it in C and embedding it in the program or doing w/e it takes.
Constraints on the random numbers:

A Set of 7 numbers that can all have different bounds:

eg: [0-X1, 0-X2, 0-X3, 0-X4, 0-X5, 0-X6, 0-X7]
Currently I am generating a list of 7 numbers with random values from [0-1) then multiplying by [X1..X7]

A Set of 13 numbers that all add up to 1

Currently just generating 13 numbers then dividing by their sum

Any ideas? Would pre calculating these numbers and storing them in a file make this faster?
Thanks!

Output:
You can speed things up a bit from what mtrw posted above just by doing what you initially described (generating a bunch of random numbers and multiplying and dividing accordingly)...
Also, you probably already know this, but be sure to do the operations in-place (*=, /=, +=, etc) when working with large-ish numpy arrays.  It makes a huge difference in memory usage with large arrays, and will give a considerable speed increase, too.
In [53]: def rand_row_doubles(row_limits, num):
   ....:     ncols = len(row_limits)
   ....:     x = np.random.random((num, ncols))
   ....:     x *= row_limits
   ....:     return x
   ....:
In [59]: %timeit rand_row_doubles(np.arange(7) + 1, 1000000)
10 loops, best of 3: 187 ms per loop

As compared to:
In [66]: %timeit ManyRandDoubles(np.arange(7) + 1, 1000000)
1 loops, best of 3: 222 ms per loop

It's not a huge difference, but if you're really worried about speed, it's something.
Just to show that it's correct:
In [68]: x.max(0)
Out[68]:
array([ 0.99999991,  1.99999971,  2.99999737,  3.99999569,  4.99999836,
        5.99999114,  6.99999738])

In [69]: x.min(0)
Out[69]:
array([  4.02099599e-07,   4.41729377e-07,   4.33480302e-08,
         7.43497138e-06,   1.28446819e-05,   4.27614385e-07,
         1.34106753e-05])

Likewise, for your "rows sum to one" part...
In [70]: def rand_rows_sum_to_one(nrows, ncols):
   ....:     x = np.random.random((ncols, nrows))
   ....:     y = x.sum(axis=0)
   ....:     x /= y
   ....:     return x.T
   ....:

In [71]: %timeit rand_rows_sum_to_one(1000000, 13)
1 loops, best of 3: 455 ms per loop

In [72]: x = rand_rows_sum_to_one(1000000, 13)

In [73]: x.sum(axis=1)
Out[73]: array([ 1.,  1.,  1., ...,  1.,  1.,  1.])

Honestly, even if you re-implement things in C, I'm not sure you'll be able to beat numpy by much on this one...  I could be very wrong, though!


Input:
set axis limits in matplotlib pyplot I have two subplots in a figure. I want to set the axes of the second subplot such that it has the same limits as the first subplot (which changes depending on the values plotted). Can someone please help me? Here is the code:
import matplotlib.pyplot as plt

plt.figure(1, figsize = (10, 20))
## First subplot: Mean value in each period (mean over replications)
plt.subplot(211, axisbg = 'w')
plt.plot(time,meanVector[0:xMax], color = '#340B8C',
         marker = 'x', ms = 4, mec = '#87051B', markevery = (asp,
                                                             2*asp))
plt.xticks(numpy.arange(0, T+1, jump), rotation = -45)
plt.axhline(y = Results[0], color = '#299967', ls = '--')
plt.ylabel('Mean Value')
plt.xlabel('Time')
plt.grid(True)

## Second subplot: moving average for determining warm-up period
## (Welch method)
plt.subplot(212)
plt.plot(time[0:len(yBarWvector)],yBarWvector, color = '#340B8C')
plt.xticks(numpy.arange(0, T+1, jump), rotation = -45)
plt.ylabel('yBarW')
plt.xlabel('Time')
plt.xlim((0, T))
plt.grid(True)

In the second subplot, what should be the arguments for plt.ylim() function? I tried defining
ymin, ymax = plt.ylim()

in the first subplot and then set
plt.ylim((ymin,ymax))

in the second subplot. But that did not work, because the returned value ymax is the maximum value taken by the y variable (mean value) in the first subplot and not the upper limit of the y-axis.
Thanks in advance.

Output:
I searched some more on the matplotlib website and figured a way to do it. If anyone has a better way, please let me know.
In the first subplot replace plt.subplot(211, axisbg = 'w') by ax1 = plt.subplot(211, axisbg = 'w')
. Then, in the second subplot, add the arguments sharex = ax1 and sharey = ax1 to the subplot command. That is, the second subplot command will now look:
plt.subplot(212, sharex = ax1, sharey = ax1)

This solves the problem. But if there are other better alternatives, please let me know.


Input:
Proving the primality of strong probable primes Using the probabilistic version of the Miller-Rabin test, I have generated a list of medium-large (200-300 digit) probable primes. But probable ain't good enough! I need to know these numbers are prime. Is there a library -- preferably wrapped or wrappable in Python -- that implements one of the more efficient primality proving algorithms?
Alternatively, does anyone know where I can find a clear, detailed, and complete description of ECPP (or a similarly fast algorithm) that does not assume a great deal of prior knowledge?
Update: I've found a Java implementation of another test, APRT-CLE, that conclusively proves primality. It verified a 291-digit prime candidate in under 10 minutes on an atom processor. Still hoping for something faster, but this seems like a promising start.

Output: As an algorithm that gives a reliable polynomial primality test, consider AKS. There is an older SO article referencing implementations and presentations of the algorithm.


Input:
Are there technical reasons a Ruby DSL like RSpec couldn't be rewritten in Python? The section below goes into more detail, but basically someone stated that the Ruby-written DSL RSpec couldn't be rewritten in Python. Is that true? If so, why?
I'm wanting to better understand the technical differences between Ruby and Python.
Update: Why am I asking this question?
The Running away from RSpec discussion has some statements about it being "impossible" to recreate RSpec in Python. I was trying to make the question a little broader in hopes of learning more of the technical differences between Ruby and Python. In hindsight, maybe I should have tightened the question's scope to just asking if it truly is impossible to recreate RSpec in Python, and if so why.
Below are just a few quotes from the Running away from RSpec discussion.
Initial Question

For the past few weeks I have been thinking a lot about RSpec and why there is no clear, definite answer when someone asks:

"I'm looking for a Python equivalent of RSpec. Where can I find such a
    thing?"

Probably the most common (and understandable) answer is that Python syntax
  wouldn't allow such a thing whereas in Ruby it is possible.

First Response to Initial Question

Not syntax exactly. Rspec monkeypatches every object inside of its
  scope, inserting the methods "should" and "should_not". You can do
  something in python, but you can't monkeypatch the built-in types.

Another Response

As you suggest, it's impossible. Mote and PySpec are just fancy ways
  to name your tests: weak implementations of one tiny corner of RSpec.
  Mote uses horrible settrace magic; PySpec adds a bunch of
  domain-irrelevant noise. Neither even supports arbitrary context
  strings. RSpec is more terse, more expressive, removes the noise, and
  is an entirely reasonable thing to build in Ruby.
That last point is important: it's not just that RSpec is possible in
  Ruby; it's actually idiomatic.

Output:
If I had to point out one great difficulty for creating a Python RSpec, it would be the lack of a good syntax in Python for creating anonymous functions (as in JavaScript) or blocks (as in Ruby). The only option for a Python programmer is to use lambdas, which is not an option at all because lambdas just accept one expression. The do ... end blocks used in RSpec would have to be written as a function before calling describe and it, as in the example below:
def should_do_stuff():
    # ...
it("should do stuff", should_do_stuff)

Not so sexy, right?
There are some difficulties in creating the should methods, but I bet it would be a smaller problem. Actually, one does not even need to use such an unusual syntaxâyou could get similar results (maybe even better, depending on your taste) using the Jasmine syntax, which can be trivially implemented.
That said, I feel that Python syntax is more focused on efficiently representing the usual program components such as classes, functions, variables, etc. It is not well suited to be extended. I, for one, think that a good Python program is one where I can see objects, and functions, and variables, and I understand what each one of these elements do. Ruby programmers, OTOH, seem to seek for a more prose-like style, where a new language is defined for a new problem. It is a good way of doing things, too, but not a Pythonic way. Python is good to represent algorithms, not prose.
Sometimes it is a draconian limit. How could one use BDD for example? Well, the usual way of pushing these limits in Python is to effectively write your own DSL, but it should REALLY be another language. That is what Pyccuracy is, for example: another language for BDD. A more mainstream example is doctest. (Actually, if I would write some BDD Python library, I would write it based on doctest.) Another example of Python DSL is Twill. And yet another example is reStructuredText, used in Sphinx.
Summarizing: IMHO the hardest barrier to DSLs in Python is the lack of a flexible syntax for creating anonymous functions. And it is not a fault*: Python is not fond of having its syntax heavily explored anywayâit is considered to make code less clear in the Python universe. If you want a new syntax in Python you are well advised to write your own language, or at least it is the way I feel.
* Or maybe it is - I have to confess that I miss anonymous functions. However, I recognize that they would be hard to implement elegantly given the Python semantic indentation.


Input:
Statically Typed Metaprogramming? I've been thinking about what I would miss in porting some Python code to a statically typed language such as F# or Scala; the libraries can be substituted, the conciseness is comparable, but I have lots of python code which is as follows:
@specialclass
class Thing(object):
    @specialFunc
    def method1(arg1, arg2):
        ...
    @specialFunc
    def method2(arg3, arg4, arg5):
        ...

Where the decorators do a huge amount: replacing the methods with callable objects with state, augmenting the class with additional data and properties, etc.. Although Python allows dynamic monkey-patch metaprogramming anywhere, anytime, by anyone, I find that essentially all my metaprogramming is done in a separate "phase" of the program. i.e.:
load/compile .py files
transform using decorators
// maybe transform a few more times using decorators
execute code // no more transformations!

These phases are basically completely distinct; I do not run any application level code in the decorators, nor do I perform any ninja replace-class-with-other-class or replace-function-with-other-function in the main application code. Although the "dynamic"ness of the language says I can do so anywhere I want, I never go around replacing functions or redefining classes in the main application code because it gets crazy very quickly.
I am, essentially, performing a single re-compile on the code before i start running it.
The only similar metapogramming i know of in statically typed languages is reflection: i.e. getting functions/classes from strings, invoking methods using argument arrays, etc. However, this basically converts the statically typed language into a dynamically typed language, losing all type safety (correct me if i'm wrong?). Ideally, I think, I would have something like the following:
load/parse application files
load/compile transformer
transform application files using transformer
compile
execute code

Essentially, you would be augmenting the compilation process with arbitrary code, compiled using the normal compiler, that will perform transformations on the main application code. The point is that it essentially emulates the "load, transform(s), execute" workflow while strictly maintaining type safety.
If the application code are borked the compiler will complain, if the transformer code is borked the compiler will complain, if the transformer code compiles but doesn't do the right thing, either it will crash or the compilation step after will complain that the final types don't add up. In any case, you will never get the runtime type-errors possible by using reflection to do dynamic dispatch: it would all be statically checked at every step.
So my question is, is this possible? Has it already been done in some language or framework which I do not know about? Is it theoretically impossible? I'm not very familiar with compiler or formal language theory, I know it would make the compilation step turing complete and with no guarantee of termination, but it seems to me that this is what I would need to match the sort of convenient code-transformation i get in a dynamic language while maintaining static type checking.
EDIT: One example use case would be a completely generic caching decorator. In python it would be:
cacheDict = {}
def cache(func):
    @functools.wraps(func)
    def wrapped(*args, **kwargs):
        cachekey = hash((args, kwargs))
        if cachekey not in cacheDict.keys():
            cacheDict[cachekey] = func(*args, **kwargs)
        return cacheDict[cachekey]
    return wrapped

@cache
def expensivepurefunction(arg1, arg2):
    # do stuff
    return result

While higher order functions can do some of this or objects-with-functions-inside can do some of this, AFAIK they cannot be generalized to work with any function taking an arbitrary set of parameters and returning an arbitrary type while maintaining type safety. I could do stuff like:
public Thingy wrap(Object O){ //this probably won't compile, but you get the idea
    return (params Object[] args) => {
        //check cache
        return InvokeWithReflection(O, args)
    }
}

But all the casting completely kills type safety.
EDIT: This is a simple example, where the function signature does not change. Ideally what I am looking for could modify the function signature, changing the input parameters or output type (a.l.a. function composition) while still maintaining type checking.

Output:
Very interesting question.
Some points regarding metaprogramming in Scala:

In scala 2.10 there will be developments in scala reflection
There is work in source to source transformation (macros) which is something you are looking for: scalamacros.org
Java has introspection (through the reflection api) but does not allow self modification. However you can use tools to support this (such as javassist). In theory you could use these tools in Scala to achieve more than introspection.
From what I could understand of your development process, you separate your domain code from your decorators (or a cross cutting concern if you will) which allow to achieve modularity and code simplicity. This can be a good use for aspect oriented programming, which allows to just that. For Java theres is a library (aspectJ), however I'm dubious it will run with Scala.


Input:
Use datetime.strftime() on years before 1900? ("require year >= 1900") I used :
utctime = datetime.datetime(1601,1,1) + datetime.timedelta(microseconds = tup[5])
 last_visit_time = "Last visit time:"+ utctime.strftime('%Y-%m-%d %H:%M:%S')
But I have the time of 1601, so the error show:
ValueError: year=1601 is before 1900; the datetime strftime() methods require year >= 1900
I used python2.7, how can I make it? Thanks a lot!

Output:
You can do the following:
>>> utctime.isoformat()
'1601-01-01T00:00:00.000050'

Now if you want to have exactly the same format as above:
iso = utctime.isoformat()
tokens = iso.strip().split("T")
last_visit_time = "Last visit time: %s %s" % (tokens[0], tokens[1].strip().split(".")[0])

Not that there seems to be a patch for strftime to fix this behavior here (not tested)


Input:
How do I create a simple pdf file in python? I'm looking for a way to output a VERY simple pdf file from Python. Basically it will consist of two columns of words, one in Russian (so utf-8 characters) and the other in English.
I've been googling for about an hour, and the packages I've found are either massive overkill (and still don't provide useful examples) such as ReportLab, or seem to assume that the only thing anyone would ever do with pdfs is concatenate several of them together (PyPdf, pdfrw).
Maybe I'm just missing something obvious, but all the ones I've seen seem to launch into some massive discussion about taking 17 pdf files and converting them into a 60 foot wide poster with 23 panes (slight exaggeration maybe), and leave me wondering how to get the "Hello World" program working. Any help would be appreciated.

Output:
You may use wkhtmltopdf. It is a command line utility that uses Webkit to convert html to pdf.
You can generate your data as html and style it with css if you want, then, use wkhtmltopdf to generate the pdf file.


Input:
Resizing numpy.memmap arrays I'm working with a bunch of large numpy arrays, and as these started to chew up too much memory lately, I wanted to replace them with numpy.memmap instances. The problem is, now and then I have to resize the arrays, and I'd preferably do that inplace. This worked quite well with ordinary arrays, but trying that on memmaps complains, that the data might be shared, and even disabling the refcheck does not help.
a = np.arange(10)
a.resize(20)
a
>>> array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])

a = np.memmap('bla.bin', dtype=int)
a
>>> memmap([0, 0, 0, 0, 0, 0, 0, 0, 0, 0])

a.resize(20, refcheck=False)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-41-f1546111a7a1> in <module>()
----> 1 a.resize(20, refcheck=False)

ValueError: cannot resize this array: it does not own its data

Resizing the underlying mmap buffer works perfectly fine. The problem is how to reflect these changes to the array object. I've seen this workaround, but unfortunately it doesn't resize the array in place. There is also some numpy documentation about resizing mmaps, but it's clearly not working, at least with version 1.8.0. Any other ideas, how to override the inbuilt resizing checks?

Output:
The issue is that the flag OWNDATA is False when you create your array. You can change that by requiring the flag to be True when you create the array:
>>> a = np.require(np.memmap('bla.bin', dtype=int), requirements=['O'])
>>> a.shape
(10,)
>>> a.flags
  C_CONTIGUOUS : True
  F_CONTIGUOUS : True
  OWNDATA : True
  WRITEABLE : True
  ALIGNED : True
  UPDATEIFCOPY : False
>>> a.resize(20, refcheck=False)
>>> a.shape
(20,)

The only caveat is that it may create the array and make a copy to be sure the requirements are met.
Edit to address saving:
If you want to save the re-sized array to disk, you can save the memmap as a .npy formatted file and open as a numpy.memmap when you need to re-open it and use as a memmap:
>>> a[9] = 1
>>> np.save('bla.npy',a)
>>> b = np.lib.format.open_memmap('bla.npy', dtype=int, mode='r+')
>>> b
memmap([0, 9, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])

Edit to offer another method:
You may get close to what you're looking for by re-sizing the base mmap (a.base or a._mmap, stored in uint8 format) and "reloading" the memmap:
>>> a = np.memmap('bla.bin', dtype=int)
>>> a
memmap([0, 0, 0, 0, 0, 0, 0, 0, 0, 0])
>>> a[3] = 7
>>> a
memmap([0, 0, 0, 7, 0, 0, 0, 0, 0, 0])
>>> a.flush()
>>> a = np.memmap('bla.bin', dtype=int)
>>> a
memmap([0, 0, 0, 7, 0, 0, 0, 0, 0, 0])
>>> a.base.resize(20*8)
>>> a.flush()
>>> a = np.memmap('bla.bin', dtype=int)
>>> a
memmap([0, 0, 0, 7, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])


Input:
What's the deal with Python 3.4, Unicode, different languages and Windows? Happy examples:
#!/usr/bin/env python
# -*- coding: utf-8 -*-

czech = u'LeoÅ¡ JanÃ¡Äek'.encode("utf-8")
print(czech)

pl = u'ZdzisÅaw BeksiÅski'.encode("utf-8")
print(pl)

jp = u'ãªã³ã° å±±æ è²å'.encode("utf-8")
print(jp)

chinese = u'äºè¡'.encode("utf-8")
print(chinese)

MIR = u'ÐÐ°ÑÐ¸Ð½Ð° Ð´Ð»Ñ ÐÐ½Ð¶ÐµÐ½ÐµÑÐ½ÑÑ Ð Ð°ÑÑÑÑÐ¾Ð²'.encode("utf-8")
print(MIR)

pt = u'Minha LÃngua Portuguesa: Ã§Ã¡Ã '.encode("utf-8")
print(pt)

Unhappy output:

And if I print them like this:
jp = u'ãªã³ã° å±±æ è²å'
print(jp)

I get:

I've also tried the following from this question (And other alternatives that involve sys.stdout.encoding):
#!/usr/bin/env python
# -*- coding: utf-8 -*-

from __future__ import print_function
import sys

def safeprint(s):
    try:
        print(s)
    except UnicodeEncodeError:
        if sys.version_info >= (3,):
            print(s.encode('utf8').decode(sys.stdout.encoding))
        else:
            print(s.encode('utf8'))

jp = u'ãªã³ã° å±±æ è²å'
safeprint(jp)

And things get even more cryptic:

And the docs were not very helpful.
So, what's the deal with Python 3.4, Unicode, different languages and Windows? Almost all possible examples I could find, deal with Python 2.x.
Is there a general and cross-platform way of printing ANY Unicode character from any language in a decent and non-nasty way in Python 3.4?
EDIT:
I've tried typing at the terminal:
chcp 65001

To change the code page, as proposed here and in the comments, and it did not work (Including the attempt with sys.stdout.encoding)

Output:
Update: Since Python 3.6, the code example that prints Unicode strings directly should just work now (even without py -mrun).

Python can print text in multiple languages in Windows console whatever chcp says:
T:\> py -mpip install win-unicode-console
T:\> py -mrun your_script.py

where your_script.py prints Unicode directly e.g.:
#!/usr/bin/env python3
print('Å¡ Ã¡Ä')      # cz
print('Å Å')       # pl
print('ãªã³ã°')     # jp
print('äºè¡')      # cn
print('Ñ Ñ Ð¶Ñ Ñ') # ru
print('Ã Ã§Ã¡Ã ')    # pt

All you need is to configure the font in your Windows console that can display the desired characters.
You could also run your Python script via IDLE without installing non-stdlib modules:
T:\> py -midlelib -r your_script.py

To write to a file/pipe, use PYTHONIOENCODING=utf-8 as @Mark Tolonen suggested:
T:\> set PYTHONIOENCODING=utf-8
T:\> py your_script.py >output-utf8.txt

Only the last solution supports non-BMP characters such as í ½í¸ (U+1F612 UNAMUSED FACE) -- py -mrun can write them but Windows console displays them as boxes even if the font supports corresponding Unicode characters (though you can copy-paste the boxes into another program, to get the characters).


Input:
Is it possible to implement Python code-completion in TextMate? PySmell seems like a good starting point.
I think it should be possible, PySmell's idehelper.py does a majority of the complex stuff, it should just be a case of giving it the current line, offering up the completions (the bit I am not sure about) and then replacing the line with the selected one.
>>> import idehelper
>>> # The path is where my PYSMELLTAGS file is located:
>>> PYSMELLDICT = idehelper.findPYSMELLDICT("/Users/dbr/Desktop/pysmell/")
>>> options = idehelper.detectCompletionType("", "" 1, 2, "", PYSMELLDICT)
>>> completions = idehelper.findCompletions("proc", PYSMELLDICT, options)
>>> print completions
[{'dup': '1', 'menu': 'pysmell.pysmell', 'kind': 'f', 'word': 'process', 'abbr': 'process(argList, excluded, output, verbose=False)'}]

It'll never be perfect, but it would be extremely useful (even if just for completing the stdlib modules, which should never change, so you wont have to constantly regenerate the PYSMELLTAGS file whenever you add a function)

Progressing! I have the utter-basics of completion in place - barely works, but it's close..
I ran python pysmells.py /System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/*.py -O /Library/Python/2.5/site-packages/pysmell/PYSMELLTAGS
Place the following in a TextMate bundle script, set "input: entire document", "output: insert as text", "activation: key equivalent: alt+esc", "scope selector: source.python"
#!/usr/bin/env python

import os
import sys
from pysmell import idehelper

CUR_WORD = os.environ.get("TM_CURRENT_WORD")

cur_file = os.environ.get("TM_FILEPATH")
orig_source = sys.stdin.read()
line_no = int(os.environ.get("TM_LINE_NUMBER"))
cur_col = int(os.environ.get("TM_LINE_INDEX"))

# PYSMELLS is currently in site-packages/pysmell/
PYSMELLDICT = idehelper.findPYSMELLDICT("/Library/Python/2.5/site-packages/pysmell/blah")
options = idehelper.detectCompletionType(cur_file, orig_source, line_no, cur_col, "", PYSMELLDICT)
completions = idehelper.findCompletions(CUR_WORD, PYSMELLDICT, options)

if len(completions) > 0:
    new_word = completions[0]['word']
    new_word = new_word.replace(CUR_WORD, "", 1) # remove what user has already typed
    print new_word

Then I made a new python document, typed "import urll" and hit alt+escape, and it completed it to "import urllib"!
As I said, it's entirely a work-in-progress, so don't use it yet..

Last update:
orestis has integrated this into the PySmell project's code! Any further fiddling will happen on github

Output:
EDIT: I've actually took your code above and integrated into a command. It will properly show a completion list for you to choose.
You can grab it here: http://github.com/orestis/pysmell/tree/master (hit download and do python setup.py install). It's rough but it works. - please report any errors on http://code.google.com/p/pysmell/
--
Hi, I'm the developer of PySmell. I also use a Mac, so if you can send me an email (contact info is in the source code) with your progress so far, I can try to integrate it :)
Oh BTW it's called PySmell - no trailing 's' :)


Input:
Is it possible to programmatically construct a Python stack frame and start execution at an arbitrary point in the code? Is it possible to programmatically construct a stack (one or more stack frames) in CPython and start execution at an arbitrary code point?  Imagine the following scenario:

You have a workflow engine where workflows can be scripted in Python with some constructs (e.g. branching, waiting/joining) that are calls to the workflow engine.
A blocking call, such as a wait or join sets up a listener condition in an event-dispatching engine with a persistent backing store of some sort.
You have a workflow script, which calls the Wait condition in the engine, waiting for some condition that will be signalled later.  This sets up the listener in the event dispatching engine.
The workflow script's state, relevant stack frames including the program counter (or equivalent state) are persisted - as the wait condition could occur days or months later.
In the interim, the workflow engine might be stopped and re-started, meaning that it must be possible to programmatically store and reconstruct the context of the workflow script.
The event dispatching engine fires the event that the wait condition picks up.
The workflow engine reads the serialised state and stack and reconstructs a thread with the stack.  It then continues execution at the point where the wait service was called.

The Question
Can this be done with an unmodified Python interpreter?  Even better, can anyone point me to some documentation that might cover this sort of thing or an example of code that programmatically constructs a stack frame and starts execution somewhere in the middle of a block of code?
Edit: To clarify 'unmodified python interpreter', I don't mind using the C API (is there enough information in a PyThreadState to do this?) but I don't want to go poking around the internals of the Python interpreter and having to build a modified one.
Update: From some initial investigation, one can get the execution context with PyThreadState_Get().  This returns the thread state in a PyThreadState (defined in pystate.h), which has a reference to the stack frame in frame.  A stack frame is held in a struct typedef'd to PyFrameObject, which is defined in frameobject.h.  PyFrameObject has a field f_lasti (props to bobince) which has a program counter expressed as an offset from the beginning of the code block.
This last is sort of good news, because it means that as long as you preserve the actual compiled code block, you should be able to reconstruct locals for as many stack frames as necessary and re-start the code.  I'd say this means that it is theoretically possible without having to make a modified python interpereter, although it means that the code is still probably going to be fiddly and tightly coupled to specific versions of the interpreter.
The three remaining problems are:

Transaction state and 'saga' rollback, which can probably be accomplished by the sort of metaclass hacking one would use to build an O/R mapper.  I did build a prototype once, so I have a fair idea of how this might be accomplished.
Robustly serialising transaction state and arbitrary locals.  This might be accomplished by reading __locals__ (which is available from the stack frame) and programatically constructing a call to pickle.  However, I don't know what, if any, gotchas there might be here.
Versioning and upgrade of workflows.  This is somewhat trickier, as the system is not providing any symbolic anchors for workflow nodes.  All we have is the anchor
In order to do this, one would have to identify the offsets of all of the entry points and map them to the new version.  Probably feasible to do manually, but I suspect it would be hard to automate.  This is probably the biggest obstacle if you want to support this capability.

Update 2: PyCodeObject (code.h) has a list of addr (f_lasti)-> line number mappings in PyCodeObject.co_lnotab (correct me if wrong here).  This might be used to facilitate a migration process to update workflows to a new version, as frozen instruction pointers could be mapped to the appropriate place in the new script, done in terms of the line numbers.  Still quite messy but a little more promising.
Update 3: I think the answer to this might be Stackless Python.  You can suspend tasks and serialise them.  I haven't worked out whether this will also work with the stack as well.

Output:
The expat python bindings included in the normal Python distribution is constructing stack frames programtically. Be warned though, it relies on undocumented and private APIs.
http://svn.python.org/view/python/trunk/Modules/pyexpat.c?rev=64048&view=auto


Input:
python single configuration file I am developing a project that requires a single configuration file whose data is used by multiple modules.
My question is: what is the common approach to that? should i read the configuration file from each
of my modules (files) or is there any other way to do it?
I was thinking to have a module named config.py that reads the configuration files and whenever I need a config I do import config and then do something like config.data['teamsdir'] get the 'teamsdir' property (for example).
response: opted for the conf.py approach then since it it is modular, flexible and simple
I can just put the configuration data directly in the file, latter if i want to read from a json file a xml file or multiple sources i just change the conf.py and make sure the data is accessed the same way.
accepted answer: chose "Alex Martelli" response because it was the most complete. voted up other answers because they where good and useful too.

Output:
I like the approach of a single config.py module whose body (when first imported) parses one or more configuration-data files and sets its own "global variables" appropriately -- though I'd favor config.teamdata over the round-about config.data['teamdata'] approach.
This assumes configuration settings are read-only once loaded (except maybe in unit-testing scenarios, where the test-code will be doing its own artificial setting of config variables to properly exercise the code-under-test) -- it basically exploits the nature of a module as the simplest Pythonic form of "singleton" (when you don't need subclassing or other features supported only by classes and not by modules, of course).
"One or more" configuration files (e.g. first one somewhere in /etc for general default settings, then one under /usr/local for site-specific overrides thereof, then again possibly one in the user's home directory for user specific settings) is a common and useful pattern.


Input: Getting CPU temperature using Python? How do I retrieve the temperature of my CPU using Python?  (Assuming I'm on Linux)

Output:
There is a newer API (see also LWN article and Linux kernel doc) showing temperatures under e.g.
/sys/class/thermal/thermal_zone0/temp

Readings are in thousandths of degrees Celcius (although in older kernels, it may have just been degrees C).


Input:
Which Python IDE can run my script line-by-line? I wouldn't call myself programmer, but I've started learning Python recently and really enjoy it.
I mainly use it for small tasks so far - scripting, text processing, KML generation and ArcGIS.
From my experience with R (working with excellent Notepad++ and NppToR combo) I usually try to work with my scripts line by line (or region by region) in order to understand what each step of my script is doing.. and to check results on the fly.
My question: is there and IDE (or editor?) for Windows that lets you evaluate single line of Python script?
I have seen quite a lot of discussion regarding IDEs in Python context.. but havent stubled upon this specific question so far.
Thanks for help!

Output: If you like R's layout. I highly recommend trying out Spyder. If you are using windows, try out Python(x,y). It is a package with a few different editors and a lot of common extra modules like scipy and numpy.


Input:
cx_freeze python single file? I've been using cx_freeze for a while now and there is one thing I've really wanted to do: put ALL of the files into ONE executable that I can distribute. It's not really user friendly to send around a folder filled with 30 files, all in the same directory. How can I accomplish this?
Thanks.

Output:
It is not supported in cx_freeze. There was discussion about it on mailing list. As I recall, the author stated that others (PyInstaller, Py2Exe) use some dirty hacks to achieve that. Some anti-virus programs can consider that behavior as a virus also.
I used PyInstaller for some time, but went back to cx_freeze because of Python 2.7 support. I pack everything using Inno Setup, so it is easy to distribute. Depends if you use Windows or not.


Input:
Python: in-memory object database which supports indexing? I'm doing some data munging which would be quite a bit simpler if I could stick a bunch of dictionaries in an in-memory database, then run simply queries against it.
For example, something like:
people = db([
    {"name": "Joe", "age": 16},
    {"name": "Jane", "favourite_color": "red"},
])
over_16 = db.filter(age__gt=16)
with_favorite_colors = db.filter(favorite_color__exists=True)

There are three confounding factors, though:

Some of the values will be Python objects, and serializing them is out of the question (too slow, breaks identity). Of course, I could work around this (eg, by storing all the items in a big list, then serializing their indexes in that listâ¦ But that could take a fair bit of fiddling).
There will be thousands of data, and I will be running lookup-heavy operations (like graph traversals) against them, so it must be possible to perform efficient (ie, indexed) queries.
As in the example, the data is unstructured, so systems which require me to predefine a schema would be tricky.

So, does such a thing exist? Or will I need to kludge something together?

Output:
What about using an in-memory SQLite database via the sqlite3 standard library module, using the special value :memory: for the connection? If you don't want to write your on SQL statements, you can always use an ORM, like SQLAlchemy, to access an in-memory SQLite database.
EDIT: I noticed you stated that the values may be Python objects, and also that you require avoiding serialization. Requiring arbitrary Python objects be stored in a database also necessitates serialization.
Can I propose a practical solution if you must keep those two requirements? Why not just use Python dictionaries as indices into your collection of Python dictionaries? It sounds like you will have idiosyncratic needs for building each of your indices; figure out what values you're going to query on, then write a function to generate and index for each. The possible values for one key in your list of dicts will be the keys for an index; the values of the index will be a list of dictionaries. Query the index by giving the value you're looking for as the key.
import collections
import itertools

def make_indices(dicts):
    color_index = collections.defaultdict(list)
    age_index = collections.defaultdict(list)
    for d in dicts:
        if 'favorite_color' in d:
            color_index[d['favorite_color']].append(d)
        if 'age' in d:
            age_index[d['age']].append(d)
    return color_index, age_index

def make_data_dicts():
    ...

data_dicts = make_data_dicts()
color_index, age_index = make_indices(data_dicts)
# Query for those with a favorite color is simply values
with_color_dicts = list(
        itertools.chain.from_iterable(color_index.values()))
# Query for people over 16
over_16 = list(
        itertools.chain.from_iterable(
            v for k, v in age_index.items() if age > 16)
)


Input:
Why are there no Makefiles for automation in Python projects? As a long time Python programmer, I wonder, if a central aspect of Python culture eluded me a long time: What do we do instead of Makefiles?
Most ruby-projects I've seen (not just rails) use Rake, shortly after node.js became popular, there was cake. In many other (not only compliled languages) there are classic Make files.
But in Python, noone seems to need such infrastructure. I randomly picked Python projects on GitHub, and they had no automation, besides the installtion, provided by setup.py.
What's the reason behind this?
Is there nothing to automate? Do most programmers prefer to run stylechecks, tests, etc. manually?
Some examples:

dependecies sets up a virtualenv and installs the dependencies
check calls the pep8 and pylint commandlinetools.
the test task depends on dependencies enables the virtualenv, starts selenium-server for the integration tests, and calls nosetest
the coffeescript task compiles all coffeescripts to minified javascript
the runserver task depends on dependencies and coffeescript
the deploy task depends on check and test and deploys the project.
the docs task calls sphinx with the appropiate arguments

Some of them are just one or two-liners, but imho, they add up. Due to the Makefile, i don't have to remember them.
To clarify: I'm not looking for a Python equivalent for Rake. I'm glad with paver. I'm looking for the reasons.

Output:
Setuptools can automate a lot of things, and for things that aren't built-in, it's easily extensible.

To run unittests, you can use the setup.py test command after having added a test_suite argument to the setup() call. (documentation)
Dependencies  (even if not available on PyPI) can be handled by adding a install_requires/extras_require/dependency_links argument to the setup() call. (documentation)
To create a .deb package, you can use the stdeb module.
For everything else, you can add custom setup.py commands.

But I agree with S.Lott, most of the tasks you'd wish to automate (except dependencies handling maybe, it's the only one I find really useful) are tasks you don't run everyday, so there wouldn't be any real productivity improvement by automating them.


Input:
Python: what are the nearest Linux and OSX equivalents of winsound.Beep? If one wishes to beep the speaker on Windows, Python 2 apparently provides a useful function: winsound.Beep(). The neat thing about this function is that it takes arguments specifying the exact frequency and duration of the beep. This is exactly what I want to do, except that I don't use Windows. So...
What are the nearest equivalents of winsound.Beep() for Linux and OSX, bringing in as few dependencies as possible?
Please note that I want to be able to beep the speaker directly, not to play a sound file. Also, I need to be able to control the frequency and duration of the beep, so curses.beep() and print '\a' won't do. Lastly, I am aware that PyGame provides extensive sound capabilities, but given that I don't require any of PyGame's other functionality, that would seem like using a sledgehammer to crack a nut (and anyway, I'm trying to do away with dependencies as far as possible).

Output:
I found a potential solution here:
http://bytes.com/topic/python/answers/25217-beeping-under-linux
It involves writing directly to /dev/audio. Not sure how portable it is or if it even works at all - i'm not on a linux machine atm.
def beep(frequency, amplitude, duration):
    sample = 8000
    half_period = int(sample/frequency/2)
    beep = chr(amplitude)*half_period+chr(0)*half_period
    beep *= int(duration*frequency)
    audio = file('/dev/audio', 'wb')
    audio.write(beep)
    audio.close()


Input:
What could affect Python string comparison performance for strings over 64 characters? I'm trying to evaluate if comparing two string get slower as their length increases. My calculations suggest comparing strings should take an amortized constant time, but my Python experiments yield strange results:
Here is a plot of string length (1 to 400) versus time in milliseconds. Automatic garbage collection is disabled, and gc.collect is run between every iteration.

I'm comparing 1 million random strings each time, counting matches as follows.The process is repeated 50 times before taking the min of all measured times.
for index in range(COUNT):
    if v1[index] == v2[index]:
        matches += 1
    else:
        non_matches += 1

What might account for the sudden increase around length 64?
Note: The following snippet can be used to try to reproduce the problem assuming v1 and v2 are two lists of random strings of length n and COUNT is their length.
timeit.timeit("for i in range(COUNT): v1[i] == v2[i]",
  "from __main__ import COUNT, v1, v2", number=50)

Further note: I've made two extra tests: comparing string with is instead of == suppresses the problem completely, and the performance is about 210ms/1M comparisons.
Since interning has been mentioned, I made sure to add a white space after each string, which should prevent interning; that doesn't change anything. Is it something else than interning then?

Output:
Python can 'intern' short strings; stores them in a special cache, and re-uses string objects from that cache.
When then comparing strings, it'll first test if it is the same pointer (e.g. an interned string):
if (a == b) {
    switch (op) {
    case Py_EQ:case Py_LE:case Py_GE:
        result = Py_True;
        goto out;
// ...

Only if that pointer comparison fails does it use a size check and memcmp to compare the strings.
Interning normally only takes place for identifiers (function names, arguments, attributes, etc.) however, not for string values created at runtime.
Another possible culprit is string constants; string literals used in code are stored as constants at compile time and reused throughout; again only one object is created and identity tests are faster on those.
For string objects that are not the same, Python tests for equal length, equal first characters then uses the memcmp() function on the internal C strings. If your strings are not interned or otherwise are reusing the same objects, all other speed characteristics come down to the memcmp() function.


Input:
How can I train a Genetic Programming algorithm onto a variable sequence of descriptors? I am currently trying to design a Genetic Programming algorithm that analyses a sequence of characters and assigns a value to those characters. Below I have made up an example set. Every line represents a data point. The values that are trained are real-valued.
Example:
For the word ABCDE the algorithm should return 1.0.
Example dataset:
ABCDE  :  1
ABCDEF  :  10
ABCDEGH  :  3
ABCDELKA  :  50
AASD  :  3
The dataset could be as large as it is needed, since this is all just made up. Lets assume the rule that the GP should figure out is not too complicated and that its explained by the data.
What i would like the algorithm to do is to approximate the values from my dataset when given the input sequence. My problem now is that each sequence can consist of a different number of characters.  I would prefer not to need to write some fancy descriptors myself, if possible.
How can I train my GP (preferably using tinyGP or python) to build this model?
Since there was so much discussion here - a diagram says a thousand words:

What I want to do is just put a data point and put that into a function. Then I get a value, which is my result. Unfortunately i do not know this function, I just have a dataset that has some examples (maybe 1000 examples just an example). Now I use the Genetic Programming Algorithm to find an Algorithm that is able to convert my Datapoint into a Result. This is my model. The problem that I have in this case is that the data points are of differing lengths. For a set length I could just specify each of the characters in the string as a input parameter. But beats me what to do if I have a varying number of input parameters.
Disclaimer: I have gotten to this problem multiple times during my studies, but we could never work out a solution that would work out well (like using a window, descriptors, etc). I would like to use a GP, because I like the technology and would like to try it out, but during Uni we also tried this with ANNs, etc, but to no avail. The problem of the variable input size remains.

Output:
Since you have not a fitness function, you will need to treat the genetic algorithm as it was a classifier. So you will need to come up with a way to evaluate a single chromosome. As others suggested you, this is a pure classification problem, not an optimization one, but, if you still want to go ahead with GA, here you have some steps to try an initial approach:
You will need:

Description of (how to encode) a valid chromosome

To work with genetic algorithms, all the solutions must have same length (there are more advanced approach with variable length enconding, but I wont enter there). So, having that, you will need to find an optimal encode method. Knowing that your input is a variable length string, you can encode your chromosome as a lookup table (dictionary in python) for your alphabet. However, a dictionary will give you some problems when you try to apply crossover or mutation operations, so is better to have the alphabet and chromosome encoding splitted. Refering to language models, you can check for n-grams, and your chromosome will have the same length as the length of your alphabet:
.. Unigrams
alphabet = "ABCDE"
chromosome1 = [1, 2, 3, 4, 5]
chromosome2 = [1, 1, 2, 1, 0]

.. Bigrams
alphabet = ["AB", "AC", "AD", "AE", "BC", "BD", "BE", "CD", "CE", "DE"]
chromosome = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

.. Trigrams
alphabet = ["ABC", "ABD", "ABE"...]
chromosome = as above, a value for each combination

2.
 Decode a chromosome to evaluate a single input
Your chromosome will represent an integer values for each element in your alphabet. So if you want to know a value of one of your inputs (variable length string) having a chromosome, you will need to try some evaluation functions, simplest one is the sum of each letter value.
alphabet = "ABC"
chromosome = [1, 2, 1]
input = "ABBBC"

# acc = accumulated value
value = reduce(lambda acc, x: acc + chromosme[alphabet.index(x)], input, 0)
# Will return ABBBC = 1+2+2+2+1 = 8

3.
 Fitness function
Your fitness function is just a simple error function. You can use simple error sum, square error... A simple evaluation function for a single gen:
def fitnessFunction(inputs, results, alphabet, chromosome):
    error = 0

    for i in range(len(inputs)):
        value = reduce(lambda acc, x: acc + chromosome[alphabet.index(x)], inputs[i], 0)
        diff = abs(results[i] - value)
        error += diff # or diff**2 if you want squared error

    return error

# A simple call -> INPUTS, EXPECTED RESULTS, ALPHABET, CURRENT CHROMOSOME
fitnessFunction(["ABC", "ABB", "ABBC"], [1,2,3], "ABC", [1, 1, 0])
# returned error will be:
# A+B+C = 1 + 1 + 0 -- expected value = 1 --> error += 1
# A+B+B = 1 + 1 + 1 -- expected value = 2 --> error += 1
# A+B+C = 1 + 1 + 1 + 0 -- expected value = 3 --> error += 0
# This chromosome has error of 2

Now, using any crossover and mutation operator you want (e.g.: one point crossover and bit flip mutation), find the chromosome that minimizes that error.
Things you can try to improve the algorithm model:

Using bigrams or trigrams
Change evaluate method (currently is a sum of lookup table values, it can be a product or something more complex)
Try using real values in chromosomes, instead of just integers


Input:
Python file open function modes I have noticed that, in addition to the documented mode characters, Python 2.7.5.1 in Windows XP and 8.1 also accepts modes U and D at least when reading files. Mode U is used in numpy's genfromtxt. Mode D has the effect that the file is deleted, as per the following code fragment:
 f = open('text.txt','rD')
 print(f.next())
 f.close()  # file text.txt is deleted when closed

Does anybody know more about these modes, especially whether they are a permanent feature of the language applicable also on Linux systems?

Output:
The D flag seems to be Windows specific. Windows seems to add several flags to the fopen function in its CRT, as described here.
While Python does filter the mode string to make sure no errors arise from it, it does allow some of the special flags, as can be seen in the Python sources here. Specifically, it seems that the N flag is filtered out, while the T and D flags are allowed:
while (*++mode) {
    if (*mode == ' ' || *mode == 'N') /* ignore spaces and N */
        continue;
    s = "+TD"; /* each of this can appear only once */
    ...

I would suggest sticking to the documented options to keep the code cross-platform.


Input:
Java equivalent of Python repr()? Is there a Java method that works like Python's repr? For example, assuming the function were named repr,
"foo\n\tbar".repr()

would return
"foo\n\tbar"
not
foo
        bar
as toString does.

Output:
In some projects, I use the following helper function to accomplish something akin to Python's repr for strings:
private static final char CONTROL_LIMIT = ' ';
private static final char PRINTABLE_LIMIT = '\u007e';
private static final char[] HEX_DIGITS = new char[] { '0', '1', '2', '3', '4', '5', '6', '7', '8', '9', 'a', 'b', 'c', 'd', 'e', 'f' };

public static String toPrintableRepresentation(String source) {

    if( source == null ) return null;
    else {

        final StringBuilder sb = new StringBuilder();
        final int limit = source.length();
        char[] hexbuf = null;

        int pointer = 0;

        sb.append('"');

        while( pointer < limit ) {

            int ch = source.charAt(pointer++);

            switch( ch ) {

            case '\0': sb.append("\\0"); break;
            case '\t': sb.append("\\t"); break;
            case '\n': sb.append("\\n"); break;
            case '\r': sb.append("\\r"); break;
            case '\"': sb.append("\\\""); break;
            case '\\': sb.append("\\\\"); break;

            default:
                if( CONTROL_LIMIT <= ch && ch <= PRINTABLE_LIMIT ) sb.append((char)ch);
                else {

                    sb.append("\\u");

                    if( hexbuf == null )
                        hexbuf = new char[4];

                    for( int offs = 4; offs > 0; ) {

                        hexbuf[--offs] = HEX_DIGITS[ch & 0xf];
                        ch >>>= 4;
                    }

                    sb.append(hexbuf, 0, 4);
                }
            }
        }

        return sb.append('"').toString();
    }
}

Its main advantage over many of the other solutions given here is, that it does not filter only a limited set of non-printable characters (like those replace-based solutions), but simply all non-printable ASCII characters. Some of it could have been written slightly nicer, but it actually does its job...
Note, that like the Python function, this one will surround the string with quotes. If you do not want that, you will have to eliminate the append('"') calls before and after the while loop.


Input:
The "right" way to add python scripting to a non-python application I'm currently in the process of adding the ability for users to extend the functionality of my desktop application (C++) using plugins scripted in python.
The naive method is easy enough. Embed the python static library and follow any number of the dozens of tutorials scattered around the web describing how to initialize and call python files, and you're pretty much done.
However...
What I'm looking for is more like what Blender does. Blender is completely customizable through python scripts, and it requires an external python executable. (Ie. python isn't actually embedded in the blender executable at all.) So, naturally, you can include any modules you already have in your site-packages directory when you are writing blender scripts. Not that that's advised, since that would limit the portability of your script.
So, what I want to know is if there is already a way to have your cake and eat it too. I want a plugin system that uses:

An embedded python interpreter.
The downside of Blender's approach is that it forces you to have a specific, possibly outdated version of python installed globally on your system. Having an embedded interpreter allows me to control what version of python is being used.
Firewall plugins.
Some equivalent of a virtualenv for each plugin; allowing them to install all the modules they need or want, but keeping them seperated from possible conflicts in other plugins. Maybe zc.buildout is a better candidate here, but, again, I'm very open to suggestion. I'm a bit at a loss as to the best way to accomplish this.
As painless as possible...
For the user. I'm willing to go the extra mile, just so long as most of the above is as transparent to the plugin writer as possible.

If any of you folks out there have any experience with this sort of thing, your help would be much appreciated. :)

Edit:
Basically, the short version of what I want is the simplicity of virtualenv, but without the bundled python interpreter, and a way to activate a specific "virtual environment" programmatically, like zc.buildout does with sys.path manipulation (the sys.path[0:0] = [...] trick).
Both virtualenv and zc.buildout contain portions of what I want, but neither produce relocatable builds that I, or a plugin developer can simply zip up and send to another computer.
Simply manipulating .pth files, or manipulating sys.path directly in a script, executed from my application gets me half-way there. But it is not enough when compiled modules are necessary, such as the PIL.

Output:
One effective way to accomplish this is to use a message-passing/communicating processes architecture, allowing you to accomplish your goal with Python, but not limiting yourself to Python.
------------------------------------
| App  <--> Ext. API <--> Protocol | <--> (Socket) <--> API.py <--> Script
------------------------------------

This diagram attempts to show the following: Your application communicates with external processes (for example Python) using message passing. This is efficient on a local machine, and can be portable because you define your own protocol. The only thing that you have to give your users is a Python library that implements your custom API, and communicates using a Send-Receive communication loop between your user's script and your application.
Define Your Application's External API
Your application's external API describes all of the functions that an external process must be able to interact with. For example, if you wish for your Python script to be able to draw a red circle in your application, your external API may include Draw(Object, Color, Position).
Define A Communication Protocol
This is the protocol that external processes use to communicate with your application through it's external API. Popular choices for this might be XML-RPC, SunRPC, JSON, or your own custom protocol and data format. The choice here needs to be sufficient for your API. For example, if you are going to be transferring binary data then JSON might require base64 encoding, while SunRPC assumes binary communication.
Build Your Application's Messaging System
This is as simple as an infinite loop receiving messages in your communication protocol, servicing the request within your application, and replying over the same socket/channel. For example, if you chose JSON then you would receive a message containing instructions to execute Draw(Object, Color, Position). After executing the request, you would reply to the request.
Build A Messaging Library For Python (or whatever else)
This is even simpler. Again, this is a loop sending and receiving messages on behalf the library user (i.e. your users writing Python scripts). The only thing this library must do is provide a programmatic interface to your Application's External API and translate requests into your communication protocol, all hidden from your users.
Using Unix Sockets, for example, will be extremely fast.
Plugin/Application Rendezvous
A common practice for discovering application plugins is to specify a "well known" directory where plugins should be placed. This might be, for example:
~/.myapp/plugins

The next step is for your application to look into this directory for plugins that exist. Your application should have a some smarts to be able to distinguish between Python scripts that are, and are not, real scripts for your application.
Let's assume that your communication protocol specifies that each script will communicate using JSON over StdInput/StdOuput. A simple, effective approach is to specify in your protocol that the first time a script runs it sends a MAGIC_ID to standard out. That is, your application reads the first, say, 8 bytes, and looks for a specific 64-bit value that identifies it as a script.
Additionally, you should include in your External API methods that allow your scripts to identify themselves. For example, a script should be able to inform the application through the External API things such as Name, Description, Capabilities, Expectations, essentially informing the application what it is, and what it will be doing.


Input:
How should I do rapid GUI development for R and Octave methods (possibly with Python)? We are a medium-sized academic research lab whose main outputs are new statistical methods for analyzing large datasets.  We generally develop in R and MATLAB/Octave.
We would like to expand the reach of our work by building simple, wizard-style user interfaces to access our methods, either web-apps like RNAfold or stand-alone applications to analyze private data.
Ideally, we would like the interfaces to do some data checking, to only use FOSS, to run in Mac and Windows environments, and to be able to generate simple charts and graphs that can be output as figures suitable for publication.  Also, we like Python because itâs a popular language in the lab and in our research community.
However, we want to be able to develop and release quickly and cheaply.  We are lucky to be able to fund one developer in the lab and s/he has to support multiple projects.
There are a lot of groups with the same needs and constraints as us, so it would be useful to be able to develop a consistent long-term strategy for this type of challenge.
Edit
I asked for comments on four possible approaches, included below with summary of comments:

Calling the R and Octave methods from Python.
Answers:

R / Python integration

Rpy  (from ChrisC)
pyRserve (from radek)
PypeR  (from radek, see answer for paper link)

Using a Python client to call in to an Octave server (from ChrisC)

Re-implementing our methods in Python.  In general, this is not easy because our methods rely on other methods also implemented in R and Octave.
Answers:

There was no comment on this approach, it's probably a bad idea.

Using a different framework to build user-interfaces.
Answers:

How about building GUIs and web-apps directly in R? (from hadley, see answer for tool refs)

Other ideas that we havenât considered, e.g., integrating with workflow management systems like Taverna.
Answers:

Try a visual programming interface like RedR (from chrisamiller)

My preferred answer
I'm first going to look into the approach suggested by Spacedman: using  QtDesigner to build the UI for PyQt4 and calling R with RPy.  We get a lot of students without a programming background who want to volunteer in the lab, and QtDesigner looks like something that we could train them to use effectively.
I would like develop GUIs directly in R as recommended by hadley but RQt, which I presume that I need to use QtDesigner, does not appear to be in active development (last update was summer of 2009).
Also, it looks like we may have to start implementing our methods exclusively in R, and translating over from MATLAB/Octave to R.  In my field, this is probably not a bad idea anyways.
Appendix:

Python libraries to simplify user interface construction:

wxPython (ChrisC)
PyQt4 (from Spacedman)

Other projects that integrate Python, R, and Octave:

SAGE (Dirk Eddelbuettel)
Shogun (Dirk Eddelbuettel)

GUIs to make GUIs:

QtDesigner with PyQt4 (from Spacedman).  Note Qt4 is both commercial and available under a LGPL.

Output: Why not continue to develop directly in R?  There are a number of packages that allow you to develop GUIs (gWidgets RGtk, tcl/tk, RQt, Rwxwidgets, rjava) or web applications.


Input:
Importing Python classes from different files in a subdirectory Here's the structure I'm working with.
directory/
          script.py
          subdir/
                 __init__.py
                 myclass01.py
                 myclass02.py

What I want to do is import in script.py the classes defined in myclass01.py and myclass02.py. If I do:
from subdir.myclass01 import *

It works fine for the class defined in myclass01.py. But with this solution if there are many classes defined in different files in subdir and I want to import all of them, I'd have to type one line for each file. There must be a shortcut for this. I tried:
from subdir.* import *

But it didn't work out.
EDIT: here are the contents of the files:
This is __init__.py (using __all__ as Apalala suggested):
__all__ = ['MyClass01','MyClass02']

This is myclass01.py:
class MyClass01:
    def printsomething():
        print 'hey'

This is myclass02.py:
class MyClass02:
    def printsomething():
        print 'sup'

This is script.py:
from subdir import *
MyClass01().printsomething()
MyClass02().printsomething()

This is the traceback that I get when I try to run script.py:
File "script.py", line 1, in <module>
    from subdir import *
AttributeError: 'module' object has no attribute 'MyClass01'

Output:
Although the names used there are different from what's shown in your question's directory structure, you could use my answer to the question titled Python namespacing and classes. The __init__.py shown there would have also allowed the usepackage.py script to have been written this way (package maps to subdir in your question, and Class1 to myclass01, etc):
from package import *

print Class1
print Class2
print Class3

Revision (updated):
Oops, sorry, the code in my other answer doesn't quite do what you want â it only automatically imports the names of any package submodules. To make it also import the named attributes from each submodule requires a few more lines of code. Here's a modified version of the package's __init__.py file (which also works in Python 3.4.1):
def _import_package_files():
    """ Dynamically import all the public attributes of the python modules
        in this file's directory (the package directory) and return a list
        of their names.
    """
    import os

    exports = []
    globals_ = globals()
    package_path = os.path.dirname(__file__)
    package_name = os.path.basename(package_path)

    for filename in os.listdir(package_path):
        modulename, ext = os.path.splitext(filename)
        if modulename[0] != '_' and ext in ('.py', '.pyw'):
            # create a package relative subpackage name
            subpackage = '{}.{}'.format(package_name, modulename)
            module = __import__(subpackage, globals_, locals_, [modulename])
            modict = module.__dict__
            names = (modict['__all__'] if '__all__' in modict else
                     [name for name in modict if name[0] != '_'])  # public names
            exports.extend(names)
            globals_.update((name, modict[name]) for name in names)

    return exports

if __name__ != '__main__':
    __all__ = ['__all__'] + _import_package_files()  # '__all__' in __all__

Alternatively you can put the above into a separate .py module file of its own in the package directory, and use it from the package's __init__.py like this:
if __name__ != '__main__':
    from ._import_package_files import *  # defines __all__
    __all__.remove('__all__')  # prevent export (optional)

Whatever you name the file, it should be something that starts with an _ underscore character so it doesn't try to import itself recursively.


Input:
Need to avoid subprocess deadlock without communicate I need a execute a command that produces a lot of output and takes a lot of time to execute (> 30 minutes). I was thinking of using subprocess.Popen to do it. I need to capture the output of the command, so I pass PIPE to stdout and stderr.
A deadlock problem when using Popen.wait() is well documented on a lot of forums, so Popen.communicate() is the proposed way of avoiding the deadlock. The problem with that solution is that communicate() blocks until the command is completed. I need to print everything that arrives on stdout while the command is executed. If there is no output after 20 minutes, the script execution will be killed.
Here are some constraints that I need to respect:

My Python version is 2.4.2 and I can't upgrade.
If the solution is still to use subprocess, I need to pass subprocess.PIPE to all std handles to avoid this bug: http://bugs.python.org/issue1124861

Is there a way to do it?

Output:
to workaround python bug #1124861 on Python2.4 you could attach stdin to a NUL device

import os
from subprocess import PIPE, STDOUT, Popen

lines = []
p = Popen(cmd, bufsize=1, stdin=open(os.devnull), stdout=PIPE, stderr=STDOUT)
for line in iter(p.stdout.readline, ''):
      print line,          # print to stdout immediately
      lines.append(line)   # capture for later
p.stdout.close()
p.wait()


Input:
Stop processing Flask route if request aborted I have a flask REST endpoint that does some cpu-intensive image processing and takes a few seconds to return. Often, this endpoint gets called, then aborted by the client. In these situations I would like to cancel processing. How can I do this in flask?
In node.js, I would do something like:
req.on('close', function(){
  //some handler
});

I was expecting flask to have something similar, or a synchronous method (request.isClosed()) that I could check at certain points during my processing and return if it's closed, but I can't find one.
I thought about sending something to test that the connection is still open, and catching the exception if it fails, but it seems Flask buffers all outputs so the exception isn't thrown until the processing completes and tries to return the result:

An established connection was aborted by the software in your host machine

How can I cancel my processing half way through if the client aborts their request?

Output:
There is a potentially... hacky solution to your problem. Flask has the ability to stream content back to the user via a generator. The hacky part would be streaming blank data as a check to see if the connection is still open and then when your content is finished the generator could produce the actual image. Your generator could check to see if processing is done and return None or "" or whatever if it's not finished.
from flask import Response

@app.route('/image')
def generate_large_image():
    def generate():
        while True:
            if not processing_finished():
                yield ""
            else:
                yield get_image()
    return Response(generate(), mimetype='image/jpeg')

I don't know what exception you'll get if the client closes the connection but I'm willing to bet its error: [Errno 32] Broken pipe


Input:
Charts in django Web Applications I want to Embed a chart in a Web Application developed using django.
I have come across Google charts API, ReportLab, PyChart, MatPlotLib and ChartDirector
I want to do it in the server side rather than send the AJAX request to Google chart APIs, as I also want to embed the chart into the PDF.
Which is the best option to use, and what are the relative merits and demerits of one over the other.

Output:
Another choice is CairoPlot.
We picked matplotlib over the others for some serious graphing inside one of our django apps, primarily because it was the only one that gave us exactly the kind of control we needed.
Performance generating PNG's was fine for us but... it was a highly specialized app with less than 10 logins a day.


Input:
Compile Matplotlib for Python on Snow Leopard I've killed half a day trying to compile matplotlib for python on Snow Leopard.  I've used the googles and found this helpful page (http://blog.hyperjeff.net/?p=160) but I still can't get it to compile.  I see comments from other users on that page, so I know I'm not alone.
I already installed zlib, libpng and freetype independently.
I edited the make.osx file to contain this at the top:
PREFIX=/usr/local

PYVERSION=2.6
PYTHON=python${PYVERSION}
ZLIBVERSION=1.2.3
PNGVERSION=1.2.33
FREETYPEVERSION=2.3.5
MACOSX_DEPLOYMENT_TARGET=10.6

## You shouldn't need to configure past this point

PKG_CONFIG_PATH="${PREFIX}/lib/pkgconfig"
CFLAGS="-Os -arch x86_64 -arch i386 -I${PREFIX}/include"
LDFLAGS="-arch x86_64 -arch i386 -L${PREFIX}/lib"
CFLAGS_DEPS="-arch i386 -arch x86_64 -I${PREFIX}/include -I${PREFIX}/include/freetype2 -isysroot /Developer/SDKs/MacOSX10.6.sdk"
LDFLAGS_DEPS="-arch i386 -arch x86_64 -L${PREFIX}/lib -syslibroot,/Developer/SDKs/MacOSX10.6.sdk"

I then run:
sudo make -f make.osx mpl_build

which gives me:
export PKG_CONFIG_PATH="/usr/local/lib/pkgconfig" &&\
    export MACOSX_DEPLOYMENT_TARGET=10.6 &&\
    export CFLAGS="-Os -arch x86_64 -arch i386 -I/usr/local/include" &&\
    export LDFLAGS="-arch x86_64 -arch i386 -L/usr/local/lib" &&\
    python2.6 setup.py build

... snip ...

gcc-4.2 -DNDEBUG -g -fwrapv -Os -Wall -Wstrict-prototypes -Os -arch x86_64 -arch i386 -I/usr/local/include -pipe -DPY_ARRAYAUNIQUE_SYMBOL=MPL_ARRAY_API -I/Library/Python/2.6/site-packages/numpy/core/include -I. -I/Library/Python/2.6/site-packages/numpy/core/include/freetype2 -I./freetype2 -I/System/Library/Frameworks/Python.framework/Versions/2.6/include/python2.6 -c src/ft2font.cpp -o build/temp.macosx-10.6-universal-2.6/src/ft2font.o
cc1plus: warning: command line option "-Wstrict-prototypes" is valid for C/ObjC but not for C++
In file included from src/ft2font.h:13,
                 from src/ft2font.cpp:1:
/usr/local/include/ft2build.h:56:38: error: freetype/config/ftheader.h: No such file or directory

... snip ...

src/ft2font.cpp:98: error: âFT_Intâ was not declared in this scope
/Library/Python/2.6/site-packages/numpy/core/include/numpy/__multiarray_api.h:1174: warning: âint _import_array()â defined but not used
lipo: can't open input file: /var/tmp//ccDOGx37.out (No such file or directory)
error: command 'gcc-4.2' failed with exit status 1
make: *** [mpl_build] Error 1

I'm just lost.

Output: According to your error message you have missing freetype headers. Can you locate them using system search functionalities. I will not lecture on using a pre-built package since I love scratching my head and compiling from the start as well.


Input:
How to setup and launch a Scrapy spider programmatically (urls and settings) I've written a working crawler using scrapy,
now I want to control it through a Django webapp, that is to say:

Set 1 or several start_urls
Set 1 or several allowed_domains
Set settings values
Start the spider
Stop / pause / resume a spider
retrieve some stats while running
retrive some stats after spider is complete.

At first I thought scrapyd was made for this, but after reading the doc, it seems that it's more a daemon able to manage 'packaged spiders', aka 'scrapy eggs'; and that all the settings (start_urls , allowed_domains, settings ) must still be hardcoded in the 'scrapy egg' itself ; so it doesn't look like a solution to my question, unless I missed something.
I also looked at this question : How to give URL to scrapy for crawling? ;
But the best answer to provide multiple urls is qualified by the author himeslf as an 'ugly hack', involving some python subprocess and complex shell handling, so I don't think the solution is to be found here. Also, it may work for start_urls, but it doesn't seem to allow allowed_domains or settings.
Then I gave a look to scrapy webservices :
It seems to be the good solution for retrieving stats. However, it still requires a running spider, and no clue to change settings
There are a several questions on this subject, none of them seems satisfactory:

using-one-scrapy-spider-for-several-websites
This one seems outdated, as scrapy has evolved a lot since 0.7
creating-a-generic-scrapy-spider
No accepted answer, still talking around tweaking shell parameters.

I know that scrapy is used in production environments ; and a tool like scrapyd shows that there are definitvely some ways to handle these requirements (I can't imagine that the scrapy eggs scrapyd is dealing with are generated by hand !)
Thanks a lot for your help.

Output:
At first I thought scrapyd was made for this, but after reading the doc, it seems that it's more a daemon able to manage 'packaged spiders', aka 'scrapy eggs'; and that all the settings (start_urls , allowed_domains, settings ) must still be hardcoded in the 'scrapy egg' itself ; so it doesn't look like a solution to my question, unless I missed something.

I don't agree to the above statement, start_urls need not be hard-coded they can be dynamically passed to the class, you should be able to pass it as an argument like this
http://localhost:6800/schedule.json -d project=myproject -d spider=somespider -d setting=DOWNLOAD_DELAY=2 -d arg1=val1

Or you should be able to retrieve the URLs from a database or a file. I get it from a database like this
class WikipediaSpider(BaseSpider):
    name = 'wikipedia'
    allowed_domains = ['wikipedia.com']
    start_urls = []

    def __init__(self, name=None, url=None, **kwargs):
        item = MovieItem()
        item['spider'] = self.name
        # You can pass a specific url to retrieve
        if url:
            if name is not None:
                self.name = name
            elif not getattr(self, 'name', None):
                raise ValueError("%s must have a name" % type(self).__name__)
            self.__dict__.update(kwargs)
            self.start_urls = [url]
        else:
            # If there is no specific URL get it from Database
            wikiliks = # < -- CODE TO RETRIEVE THE LINKS FROM DB -->
            if wikiliks == None:
                print "**************************************"
                print "No Links to Query"
                print "**************************************"
                return None

            for link in wikiliks:
                # SOME PROCESSING ON THE LINK GOES HERE
                self.start_urls.append(urllib.unquote_plus(link[0]))

    def parse(self, response):
        hxs = HtmlXPathSelector(response)
        # Remaining parse code goes here


Input:
Why does map return a map object instead of a list in Python 3? I am interested in understanding the new language design of Python 3.x.
I do enjoy, in Python 2.7, the function map:
Python 2.7.12
In[2]: map(lambda x: x+1, [1,2,3])
Out[2]: [2, 3, 4]

However, in Python 3.x things have changed:
Python 3.5.1
In[2]: map(lambda x: x+1, [1,2,3])
Out[2]: <map at 0x4218390>

I understand the how, but I could not find a reference to the why. Why did the language designers make this choice, which, in my opinion, introduces a great deal of pain. Was this to arm-wrestle developers in sticking to list comprehensions?
IMO, list can be naturally thought as Functors; and I have been somehow been thought to think in this way:
fmap :: (a -> b) -> f a -> f b

Output:
Because it returns an iterator, it omit storing the full list in the memory. So that you can easily iterate over it in the future not making pain to memory. Possibly you even don't need a full list, but the part of it, until your condition is matched.
You can find this docs useful, because iterators are awesome.

An object representing a stream of data. Repeated calls to the iteratorâs __next__() method (or passing it to the built-in function next()) return successive items in the stream. When no more data are available a StopIteration exception is raised instead. At this point, the iterator object is exhausted and any further calls to its __next__() method just raise StopIteration again. Iterators are required to have an __iter__() method that returns the iterator object itself so every iterator is also iterable and may be used in most places where other iterables are accepted. One notable exception is code which attempts multiple iteration passes. A container object (such as a list) produces a fresh new iterator each time you pass it to the iter() function or use it in a for loop. Attempting this with an iterator will just return the same exhausted iterator object used in the previous iteration pass, making it appear like an empty container.


Input:
How do you add breakpoints to a python program in IDLE? I solved this already, I'm just posting it here because I couldn't figure it out with google, or by reading the docs. I only found it by accident.
To add a breakpoint, right-click on the line that you want to add a breakpoint on and select "Set Breakpoint". The line will become highlighted. (Note that this only works when you are editing a file; it does not work at the interpreter.)
On Mac, you need to Control-click instead of right clicking.

Output:
There is an open Python issue about the lack of documentation for IDLE breakpoints.  Contributions welcome!
Update: The on-line documentation and the IDLE help files (for 2.7.4, 3.2.4, and 3.3.0) have been updated, with Nick's help.


Input:
Is there "Edit and Continue" in PyCharm? Reload code into running program like in Eclipse / PyDev? Hi all Python developers!
In Eclipse with PyDev it is possible to edit a Python file while debugging. On save, the PyDev debugger will reload the updated code into the running program and uses my new code. How can I do the same thing in JetBrains PyCharm (using Community Edition)?
Eclipse / PyDev writes an output like this when I do that:
pydev debugger: Start reloading module: "MyWidget" ...
pydev debugger: Updated function code: <function close at 0x055F4E70>
pydev debugger: reload finished

I searched settings and web and could not find any hint. Very glad about any idea. Thx.
Edit: I found out in Eclipse/PyDev one has to be in debug mode to be able to use this feature. I tested in PyCharm, but there was no reload done.

Output:
After all I found a useful and acceptable workaround for my question. It works in PyCharm Community Edition 3.1.2 and I assume it will do in commercial edition as well. I tested on a mid-scale project using Python 2.7.6, PySide (Qt) with one main window and 20+ widgets, tabs, whatever. Follow these steps...

Work in Pycharm on a python project :-)
Execute your code in Debug mode (did not tried Release so far)
Edit some code in one your modules imported during the life of your program
Make your program pause. To achieve this, you can click the "Pause" button of in PyCharms Debug view and then any place in your applications main window where it would need to do something (for example on a tab header). If you have a long a running task and no UI, you may place a breakpoint in a place your program often comes by.
In the Debug view, switch to the Console tab. There is a button on the left Show command line. Click this.
In the console, type in reload(MyModifiedModule) if this call fails, write import MyModifiedModule and try again.
Click resume in PyCharm.
Try the code you fixed.

There are some restrictions on this... It won't fix changes in your main method or main window, cause it won't be created again. In my tests I could not reload widgets from Qt. But it worked for classes like data containers or workers.
May the force be with you as you try this and do not hesitate to add your experiences.
Thank you.


Input:
Right way to return proxy model instance from a base model instance in Django? Say I have models:
class Animal(models.Model):
    type = models.CharField(max_length=255)

class Dog(Animal):
    def make_sound(self):
        print "Woof!"
    class Meta:
        proxy = True

class Cat(Animal):
    def make_sound(self):
        print "Meow!"
    class Meta:
        proxy = True

Let's say I want to do:
 animals = Animal.objects.all()
 for animal in animals:
     animal.make_sound()

I want to get back a series of Woofs and Meows. Clearly, I could just define a make_sound in the original model that forks based on animal_type, but then every time I add a new animal type (imagine they're in different apps), I'd have to go in and edit that make_sound function. I'd rather just define proxy models and have them define the behavior themselves. From what I can tell, there's no way of returning mixed Cat or Dog instances, but I figured maybe I could define a "get_proxy_model" method on the main class that returns a cat or a dog model.
Surely you could do this, and pass something like the primary key and then just do Cat.objects.get(pk = passed_in_primary_key). But that'd mean doing an extra query for data you already have which seems redundant. Is there any way to turn an animal into a cat or a dog instance in an efficient way? What's the right way to do what I want to achieve?

Output:
The Metaclass approach proposed by thedk is indeed a very powerful way to go, however, I had to combine it with an answer to the question here to have the query return a proxy model instance. The simplified version of the code adapted to the previous example would be:
from django.db.models.base import ModelBase

class InheritanceMetaclass(ModelBase):
    def __call__(cls, *args, **kwargs):
        obj = super(InheritanceMetaclass, cls).__call__(*args, **kwargs)
        return obj.get_object()

class Animal(models.Model):
    __metaclass__ = InheritanceMetaclass
    type = models.CharField(max_length=255)
    object_class = models.CharField(max_length=20)

    def save(self, *args, **kwargs):
        if not self.object_class:
            self.object_class = self._meta.module_name
        super(Animal, self).save( *args, **kwargs)

    def get_object(self):
        if self.object_class in SUBCLASSES_OF_ANIMAL:
            self.__class__ = SUBCLASSES_OF_ANIMAL[self.object_class]
        return self

class Dog(Animal):
    class Meta:
        proxy = True
    def make_sound(self):
        print "Woof!"

class Cat(Animal):
    class Meta:
        proxy = True
    def make_sound(self):
        print "Meow!"

SUBCLASSES_OF_ANIMAL = dict([(cls.__name__, cls) for cls in ANIMAL.__subclasses__()])

The advantage of this proxy approach is that no db migration is required upon creation of new subclasses. The drawback is that no specific fields can be added to the subclasses.
I would be happy to have feedback on this approach.


Input:
Use pdb.set_trace() in a script that reads stdin via a pipe I have a python script that reads stdin via a pipe, and I cannot seem to use it with pdb.set_trace().
my_script.py:
#!/usr/bin/env python
import sys
import pdb

def main():
    for line in sys.stdin:
        print "Printing a line: " +line

if __name__=='__main__':
    status = main()

Suppose tempfile.csv is some file with two lines,
$ cat tempfile.csv
line1
line2

then I can run my script with:
$ cat tempfile.csv | ./my_script.py, and everything is fine:
$ cat tempfile.csv | ./my_script.py
Printing a line:  line1

Printing a line:  line2

On the other hand, if I put pdb.set_trace() anywhere then I get an error.  For example, putting pdb.set_trace() below def main(), then I get
$ cat tempfile.csv | ./my_script.py
> /home/ilangmore/mobiuss/TM/branches/hadooprotype/my_script.py(7)main()
-> for line in sys.stdin:
(Pdb) *** NameError: name 'line1' is not defined
(Pdb) *** NameError: name 'line2' is not defined
(Pdb)
Traceback (most recent call last):
  File "./my_script.py", line 11, in <module>
    status = main()
  File "./my_script.py", line 7, in main
    for line in sys.stdin:
  File "./my_script.py", line 7, in main
    for line in sys.stdin:
  File "/usr/lib/python2.7/bdb.py", line 48, in trace_dispatch
    return self.dispatch_line(frame)
  File "/usr/lib/python2.7/bdb.py", line 67, in dispatch_line
    if self.quitting: raise BdbQuit
bdb.BdbQuit

Note that my question is probably related to this question (i.e. pdb by default reads from stdin), but I need more help.

Output:
Here's an example of what worked for me:
lines = sys.stdin.readlines()
sys.stdin = open('/dev/tty')
pdb.set_trace()


Input:
Read timeout using either urllib2 or any other http library I have code for reading an url like this:
from urllib2 import Request, urlopen
req = Request(url)
for key, val in headers.items():
    req.add_header(key, val)
res = urlopen(req, timeout = timeout)
# This line blocks
content = res.read()

The timeout works for the urlopen() call. But then the code gets to the res.read() call where I want to read the response data and the timeout isn't applied there. So the read call may hang almost forever waiting for data from the server. The only solution I've found is to use a signal to interrupt the read() which is not suitable for me since I'm using threads.
What other options are there? Is there a HTTP library for Python that handles read timeouts? I've looked at httplib2 and requests and they seem to suffer the same issue as above. I don't want to write my own nonblocking network code using the socket module because I think there should already be a library for this.
Update: None of the solutions below are doing it for me. You can see for yourself that setting the socket or urlopen timeout has no effect when downloading a large file:
from urllib2 import urlopen
url = 'http://iso.linuxquestions.org/download/388/7163/http/se.releases.ubuntu.com/ubuntu-12.04.3-desktop-i386.iso'
c = urlopen(url)
c.read()

At least on Windows with Python 2.7.3, the timeouts are being completely ignored.

Output:
I found in my tests (using the technique described here) that a timeout set in the urlopen() call also effects the read() call:
import urllib2 as u
c = u.urlopen('http://localhost/', timeout=5.0)
s = c.read(1<<20)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/socket.py", line 380, in read
    data = self._sock.recv(left)
  File "/usr/lib/python2.7/httplib.py", line 561, in read
    s = self.fp.read(amt)
  File "/usr/lib/python2.7/httplib.py", line 1298, in read
    return s + self._file.read(amt - len(s))
  File "/usr/lib/python2.7/socket.py", line 380, in read
    data = self._sock.recv(left)
socket.timeout: timed out

Maybe it's a feature of newer versions?  I'm using Python 2.7 on a 12.04 Ubuntu straight out of the box.